0110011011110110101111110100000100011100010100011111001100101110101000110001110000001100100001110101001110011 1100100011010100100011111010111100000010011010100101000000000101001000001000010001011001100101000000110010010 1111100110000100100110101010100010101001110101110010001011101100101001000001001111100011101001010010011011011 0110111001110110100101111110010100010100010001100010000110010100010000001001001111101111000011101101000100111 0000111110011011100000110100111111110000110011011100110001001110011001101110101101101011001111111100010011001 0000010011011111000010001111101110011101110101101010110011100101001011011000001001100100010110011000011111010 0101000100011010001011010010000011110100000011110110101000101111001000011101001100011011111101111100111111010 1010010001011101110111101101110010001100010101100010110010111001101100111111010111100100001011111011011010010 0011000011101100101010000011111110000101110000010010111100111011100000001111100001111001101000110100101000000 0000110111100000000110111010010000010110001101001011110100001000110110010101010001111110100100000000110101110 0100001110010001000011101101101100000101101111000000110000110101011010000111001100110001011110001010101101000 0101001111000111011010011101001101001110100010010111000001010110100001111001010000001101110110111101001010011 0111000100110000100100100001110000110010000001000010110111000010010101011111100110010101110001100010010001000 0111001111011100000010101101101101010100101111010101011010010011111111110011001000011010011011001100110011100 0000100111001000011010010111011100111000000101111111001100111111111111011100010111111100000100110011101100001 1011101001100010010000000011011110101101111101010011001111000110101101000001111011010000111100100011000110111 1101110010100011100000110011001100100000101011100101000010011101010111100100010001111001101001010001110100011 1100010110111100000101110110101100001010110010101110111010011101011010010110101010001100101101110101111000000 0100010000010000101001111101101100001101110001010011001111000000001010010110010110101001010101010001110000101 0000010110111011010101110110001001010011101010101101111110011111000101001110010001100110001101010000000011010 1001011011110001111111000100011011110110011000011010000101001100011100100110110000011110111010001000000111010 0101100110111000101100100110000011011001011011110010001000110100110010101000001110111111011110101001010101111 1101011001000110000110011100110011010010000011111100011110011010101000000010011001000110101010110101000000101 volume 1, number 1 january april 2020 e-issn: 2711-4627 paradigmplus e-issn: 2711-4627 volume 1, number 1 january april 2020 bogota, colombia https://journals.itiud.org/index.php/paradigmplus doi: https://doi.org/10.55969/paradigmplus editors hector florez id universidad distrital francisco josé de caldas bogota, colombia christian grévisse id université du luxembourg esch-sur-alzette, luxembourg fernando yepes-calderon id children’s hospital los angeles los angeles, united states this number and its articles are published under the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. corresponding editor: hector florez universidad distrital francisco josé de caldas bogota, colombia haflorezf@udistrital.edu.co https://journals.itiud.org/index.php/paradigmplus https://doi.org/10.55969/paradigmplus https://orcid.org/0000-0002-5339-4459 https://orcid.org/0000-0002-9585-1160 https://orcid.org/0000-0001-9184-787x haflorezf@udistrital.edu.co journal information paradigmplus is a peer-reviewed journal published by the iti research group1 in collaboration with science based platforms for a global audience of professionals in computer science interested in disseminating computer-based research in all knowledge areas. focus and scope. paradigmplus publishes well-developed manuscripts describing theoretical, empirical, conceptual, and experimental results in all areas of computer science. the journal will address (but will not limit itself to the following areas: algorithms and analysis of algorithms, artificial intelligence, bioinformatics, business analytics, brain-computer interface, cloud computing, computer-aided design, computer architecture, computer vision, cryptography, data analysis, data mining, data science, decision systems, distributed and parallel computing, enterprise information systems applications, gamification and serious games, geoinformatics, health care information systems, human-computer interaction, interaction design and usability, it architectures, learning management systems, mobile information processing systems, natural language, robotic autonomy, software and systems modeling, software architectures, software design engineering, and theory and formal methods. publication frequency. paradigmplus publishes one volume per year and three numbers per volume. the first number was published in april 2020. peer-review policy. paradigmplus performs a double-blind peer-review process, where the reviewers do not know the names or affiliations of the authors and the reviewer reports provided to the authors are anonymous. every submission will be reviewed at least by two experts. reviewers will also be asked to indicate how innovative and significant the research is. the editors will reach a decision based on these reports. when a submission is accepted, the editor will include the names and affiliation of the reviewers in alphabetical order by last name. open access policy. paradigmplus is open-access allowing free access to its content in order to foster the interchange of global knowledge. article processing charges. paradigmplus is completely free to both readers and authors. the manuscripts are submitted, evaluated, proofread, and published without any cost. creative commons licence. paradigmplus is licensed under a creative commons license attribution 4.0 international copyright terms. the corresponding author, acting on behalf of all of the authors must complete, print, hand-written sign, scan, and submit the author agreement to publish a contribution as openaccess on paradigmplus document, where authors: 1https://www.itiud.org/index.php?lang=en ii https://www.itiud.org/index.php?lang=en • agree that the contribution is made available publicly under the creative commons license attribution 4.0 international. • grant the editors and its archiving partners the non-exclusive and irrevocable right to archive the contribution and to make it accessible (online and free of charge) for public distribution. • agree that the author names and affiliations are part of the associated metadata and may be stored. • acknowledge that the editors hold the copyright for the journal paradigmplus. • have not included any third-party material such as figures, code, data sets, and others without having properly indicated its source and credited its original authors. • warrant that the contribution does not infringe on any rights of third parties. • understand and agree that the full responsibility for the content of the contribution rests upon the authors of the contribution. • release the journals’ editors and the archiving partners of the journal paradigmplus from any liability caused by the publication or archiving of the contribution. • have read and completely understand the conditions of the creative commons license attribution 4.0 international and agree to apply this license to the contribution. anti-plagiarism policy. every submission is screened by turnitin. submissions with a high rate of coincidences are declined by the editor without review. publication ethics and malpractice statement. paradigmplus ensures that all involved actors i.e., authors, reviewers, and editors strictly follow the international rules on ethics in the publication process. thus, paradigmplus follows the ethics published by the committee on publication ethics (cope) in their cope best practice guidelines for journal editors and the international standards for editors and authors, which includes the following aspects: • general duties and responsibilities of the editorial board. having overall responsibility for the journal, the paradigmplus editorial board are committed to: working together to meet the readers’ and authors’ needs, fostering the continuous improvement of the journal, ensuring the quality of published material, ensuring freedom of expression, maintaining the academic integrity of their content, and preventing commercial interests from compromising intellectual standards, posting corrections, clarifications, retractions, and apologies when necessary. • relations with readers. readers will be informed about research funders and their role in the research. • relations with authors. paradigmplus is committed to ensuring the quality of its published material, informing the goals and standards of the journal. the decision of editors to accept or decline a submission for publication is based on the relevance of the work, originality, and pertinence of the study regarding the journal’s focus and scope. paradigmplus includes a description of the process for peer evaluation of each work received and provides a guide for authors with this information. the guide is up to date and contains a link to the code of ethics. paradigmplus recognizes the right of authors to appeal to editorial decisions. editors will not change their decision in accepting or declining submission unless extraordinary circumstances or irregularities are detected. any change in the composition of the editorial board will not affect decisions already made, except in unusual cases where serious circumstances converge. iii • relations with reviewers. paradigmplus makes available to reviewers a guide to what is expected from them. during the review process, reviewers’ identity is protected at all times, ensuring anonymity. nevertheless, when a submission is accepted, reviewers’ information is included in the published article. • claims. paradigmplus is committed to answering quickly to complaints, ensuring a smooth process of resolution. furthermore, if a claimant is still dissatisfied, paradigmplus considers that they have the right to raise their protests to other instances. • promoting academic integrity. paradigmplus ensures that its published material conforms to internationally-accepted ethical standards. • protection of individual data. paradigmplus guarantees the confidentiality of individuals’ information. • tracking malpractice. paradigmplus accepts the obligation to act appropriately in the event of suspected malpractice or misconduct. the editors not only reject manuscripts when there are doubts about possible misconduct, but they are also considered ethically obliged to report suspected cases of misconduct. every reasonable effort is made by paradigmplus to ensure that works submitted for evaluation are rigorous in their methods and ethically appropriate. • integrity and academic rigor. whenever there is evidence that a published work contains significant misstatements, misleading, or distorted statements, it must be corrected immediately. in the event of fraudulent content being detected, the work will be removed as soon as that is known, and readers and indexing systems will be informed immediately. practices that are considered unacceptable will be reported. • conflict of interest. paradigmplus will establish the necessary mechanisms to avoid or resolve potential conflicts of interest between authors, reviewers, and the editorial board. iv editorial paradigmplus is an initiative that was born in the international conference on applied informatics (icai)2. some members of the program committee that participated in icai 2018 and icai 2019 decided to undertake this journal in order to provide a new scenario for publishing high quality articles. to achieve this, all submitted articles underwent a double blind peer review process, where each article was reviewed by at least three experts from the editorial team. paradigmplus is supported by the editorial team members, who are researchers that hold ph.d. degrees in computing areas. editorial team members come from the following countries: argentina, brazil, china, colombia, cyprus, ecuador, germany, india, japan, luxembourg, morocco, netherlands, spain, switzerland, united arab emirates, united kingdom, and united states. this number contains the articles reviewed and accepted from january to april 2020. in this period, paradigmplus received 7 submissions. after a careful discussion, the editorial board decided to accept the articles: evaluation of the bias in the management of patient’s appointments in a pediatric office, whose authors are from argentina and what is game balancing? an examination of concepts, whose authors are from germany. we thank the editorial team members for their efforts. april 2020 hector florez christian grévisse fernando yepes-calderón 2https://icai.itiud.org/ v https://icai.itiud.org/ organization editor-in-chief hector florez universidad distrital francisco josé de caldas, colombia associate editors christian grévisse université du luxembourg, luxembourg fernando yepes-calderon children’s hospital los angeles, united states reviewers in this number hüseyin bicen yakin dogu üniversitesi, cyprus jaime chavarriaga universidad de los andes, colombia mohamed salim el bazzi université ibn zohr, morocco tong li beijing university of technology, china ivan mura duke kunshan university, china maria florencia pollo-cattaneo universidad tecnológica nacional facultad regional buenos aires, argentina sweta singh savitribai phule pune university, india fernando yepes-calderon children’s hospital los angeles, united states vi table of contents evaluation of the bias in the management of patient’s appointments in a pediatric office 1 cinthia vegega, pablo pytel, maria florencia pollo-cattaneo what is game balancing? an examination of concepts 22 alexander becker, daniel görlich vii 0110011011110110101111110100000100011100010100011111001100101110101000110001110000001100100001110101001110011 1100100011010100100011111010111100000010011010100101000000000101001000001000010001011001100101000000110010010 1111100110000100100110101010100010101001110101110010001011101100101001000001001111100011101001010010011011011 0110111001110110100101111110010100010100010001100010000110010100010000001001001111101111000011101101000100111 0000111110011011100000110100111111110000110011011100110001001110011001101110101101101011001111111100010011001 0000010011011111000010001111101110011101110101101010110011100101001011011000001001100100010110011000011111010 0101000100011010001011010010000011110100000011110110101000101111001000011101001100011011111101111100111111010 1010010001011101110111101101110010001100010101100010110010111001101100111111010111100100001011111011011010010 0011000011101100101010000011111110000101110000010010111100111011100000001111100001111001101000110100101000000 0000110111100000000110111010010000010110001101001011110100001000110110010101010001111110100100000000110101110 0100001110010001000011101101101100000101101111000000110000110101011010000111001100110001011110001010101101000 0101001111000111011010011101001101001110100010010111000001010110100001111001010000001101110110111101001010011 0111000100110000100100100001110000110010000001000010110111000010010101011111100110010101110001100010010001000 0111001111011100000010101101101101010100101111010101011010010011111111110011001000011010011011001100110011100 0000100111001000011010010111011100111000000101111111001100111111111111011100010111111100000100110011101100001 1011101001100010010000000011011110101101111101010011001111000110101101000001111011010000111100100011000110111 1101110010100011100000110011001100100000101011100101000010011101010111100100010001111001101001010001110100011 1100010110111100000101110110101100001010110010101110111010011101011010010110101010001100101101110101111000000 0100010000010000101001111101101100001101110001010011001111000000001010010110010110101001010101010001110000101 0000010110111011010101110110001001010011101010101101111110011111000101001110010001100110001101010000000011010 1001011011110001111111000100011011110110011000011010000101001100011100100110110000011110111010001000000111010 0101100110111000101100100110000011011001011011110010001000110100110010101000001110111111011110101001010101111 1101011001000110000110011100110011010010000011111100011110011010101000000010011001000110101010110101000000101 volume 1, number 2 may august 2020 e-issn: 2711-4627 paradigmplus e-issn: 2711-4627 volume 1, number 2 may august 2020 bogota, colombia https://journals.itiud.org/index.php/paradigmplus doi: https://doi.org/10.55969/paradigmplus editors hector florez id universidad distrital francisco josé de caldas bogota, colombia christian grévisse id université du luxembourg esch-sur-alzette, luxembourg fernando yepes-calderon id children’s hospital los angeles los angeles, united states this number and its articles are published under the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. corresponding editor: hector florez universidad distrital francisco josé de caldas bogota, colombia haflorezf@udistrital.edu.co https://journals.itiud.org/index.php/paradigmplus https://doi.org/10.55969/paradigmplus https://orcid.org/0000-0002-5339-4459 https://orcid.org/0000-0002-9585-1160 https://orcid.org/0000-0001-9184-787x haflorezf@udistrital.edu.co journal information paradigmplus is a peer-reviewed journal published by the iti research group1 in collaboration with science based platforms for a global audience of professionals in computer science interested in disseminating computer-based research in all knowledge areas. focus and scope. paradigmplus publishes well-developed manuscripts describing theoretical, empirical, conceptual, and experimental results in all areas of computer science. the journal will address (but will not limit itself to the following areas: algorithms and analysis of algorithms, artificial intelligence, bioinformatics, business analytics, brain-computer interface, cloud computing, computer-aided design, computer architecture, computer vision, cryptography, data analysis, data mining, data science, decision systems, distributed and parallel computing, enterprise information systems applications, gamification and serious games, geoinformatics, health care information systems, human-computer interaction, interaction design and usability, it architectures, learning management systems, mobile information processing systems, natural language, robotic autonomy, software and systems modeling, software architectures, software design engineering, and theory and formal methods. publication frequency. paradigmplus publishes one volume per year and three numbers per volume. the first number was published in april 2020. peer-review policy. paradigmplus performs a double-blind peer-review process, where the reviewers do not know the names or affiliations of the authors and the reviewer reports provided to the authors are anonymous. every submission will be reviewed at least by two experts. reviewers will also be asked to indicate how innovative and significant the research is. the editors will reach a decision based on these reports. when a submission is accepted, the editor will include the names and affiliation of the reviewers in alphabetical order by last name. open access policy. paradigmplus is open-access allowing free access to its content in order to foster the interchange of global knowledge. article processing charges. paradigmplus is completely free to both readers and authors. the manuscripts are submitted, evaluated, proofread, and published without any cost. creative commons licence. paradigmplus is licensed under a creative commons license attribution 4.0 international copyright terms. the corresponding author, acting on behalf of all of the authors must complete, print, hand-written sign, scan, and submit the author agreement to publish a contribution as openaccess on paradigmplus document, where authors: 1https://www.itiud.org/index.php?lang=en ii https://www.itiud.org/index.php?lang=en • agree that the contribution is made available publicly under the creative commons license attribution 4.0 international. • grant the editors and its archiving partners the non-exclusive and irrevocable right to archive the contribution and to make it accessible (online and free of charge) for public distribution. • agree that the author names and affiliations are part of the associated metadata and may be stored. • acknowledge that the editors hold the copyright for the journal paradigmplus. • have not included any third-party material such as figures, code, data sets, and others without having properly indicated its source and credited its original authors. • warrant that the contribution does not infringe on any rights of third parties. • understand and agree that the full responsibility for the content of the contribution rests upon the authors of the contribution. • release the journals’ editors and the archiving partners of the journal paradigmplus from any liability caused by the publication or archiving of the contribution. • have read and completely understand the conditions of the creative commons license attribution 4.0 international and agree to apply this license to the contribution. anti-plagiarism policy. every submission is screened by turnitin. submissions with a high rate of coincidences are declined by the editor without review. publication ethics and malpractice statement. paradigmplus ensures that all involved actors i.e., authors, reviewers, and editors strictly follow the international rules on ethics in the publication process. thus, paradigmplus follows the ethics published by the committee on publication ethics (cope) in their cope best practice guidelines for journal editors and the international standards for editors and authors, which includes the following aspects: • general duties and responsibilities of the editorial board. having overall responsibility for the journal, the paradigmplus editorial board are committed to: working together to meet the readers’ and authors’ needs, fostering the continuous improvement of the journal, ensuring the quality of published material, ensuring freedom of expression, maintaining the academic integrity of their content, and preventing commercial interests from compromising intellectual standards, posting corrections, clarifications, retractions, and apologies when necessary. • relations with readers. readers will be informed about research funders and their role in the research. • relations with authors. paradigmplus is committed to ensuring the quality of its published material, informing the goals and standards of the journal. the decision of editors to accept or decline a submission for publication is based on the relevance of the work, originality, and pertinence of the study regarding the journal’s focus and scope. paradigmplus includes a description of the process for peer evaluation of each work received and provides a guide for authors with this information. the guide is up to date and contains a link to the code of ethics. paradigmplus recognizes the right of authors to appeal to editorial decisions. editors will not change their decision in accepting or declining submission unless extraordinary circumstances or irregularities are detected. any change in the composition of the editorial board will not affect decisions already made, except in unusual cases where serious circumstances converge. iii • relations with reviewers. paradigmplus makes available to reviewers a guide to what is expected from them. during the review process, reviewers’ identity is protected at all times, ensuring anonymity. nevertheless, when a submission is accepted, reviewers’ information is included in the published article. • claims. paradigmplus is committed to answering quickly to complaints, ensuring a smooth process of resolution. furthermore, if a claimant is still dissatisfied, paradigmplus considers that they have the right to raise their protests to other instances. • promoting academic integrity. paradigmplus ensures that its published material conforms to internationally-accepted ethical standards. • protection of individual data. paradigmplus guarantees the confidentiality of individuals’ information. • tracking malpractice. paradigmplus accepts the obligation to act appropriately in the event of suspected malpractice or misconduct. the editors not only reject manuscripts when there are doubts about possible misconduct, but they are also considered ethically obliged to report suspected cases of misconduct. every reasonable effort is made by paradigmplus to ensure that works submitted for evaluation are rigorous in their methods and ethically appropriate. • integrity and academic rigor. whenever there is evidence that a published work contains significant misstatements, misleading, or distorted statements, it must be corrected immediately. in the event of fraudulent content being detected, the work will be removed as soon as that is known, and readers and indexing systems will be informed immediately. practices that are considered unacceptable will be reported. • conflict of interest. paradigmplus will establish the necessary mechanisms to avoid or resolve potential conflicts of interest between authors, reviewers, and the editorial board. iv editorial paradigmplus is an initiative that was born in the international conference on applied informatics (icai)2. some members of the program committee that participated in icai 2018 and icai 2019 decided to undertake this journal in order to provide a new scenario for publishing high quality articles. to achieve this, all submitted articles underwent a double blind peer review process, where each article was reviewed by at least three experts from the editorial team. paradigmplus is supported by the editorial team members, who are researchers that hold ph.d. degrees in computing areas. editorial team members come from the following countries: argentina, brazil, chile, china, colombia, cyprus, ecuador, germany, greece, india, japan, luxembourg, morocco, netherlands, spain, switzerland, united arab emirates, united kingdom, and united states. this number contains the articles reviewed and accepted from may to august 2020. in this period, paradigmplus received 4 submissions. after a careful discussion, the editorial board decided to accept the articles: using analog ensembles with alternative metrics for hindcasting with multistations, whose authors are from portugal and automated fine grained traceability links recovery between high level requirements and source code implementations, whose authors are from colombia. we thank the editorial team members for their efforts. august 2020 hector florez christian grévisse fernando yepes-calderón 2https://icai.itiud.org/ v https://icai.itiud.org/ organization editor-in-chief hector florez universidad distrital francisco josé de caldas, colombia associate editors christian grévisse université du luxembourg, luxembourg fernando yepes-calderon children’s hospital los angeles, united states reviewers in this number jorge bacca fundacion universitaria konrad lorenz, colombia luis de-la-fuente-valentin universidad internacional de la rioja, spain keletso letsholo higher colleges of technology, united arab emirates tong li beijing university of technology, china diego peluffo-ordóñez universidad yachay tech, ecuador fernando yepes-calderon children’s hospital los angeles, united states vi table of contents using analog ensembles with alternative metrics for hindcasting with multistations 1 carlos balsa, c. veiga rodrigues, isabel lopes, josé rufino automated fine grained traceability links recovery between high level requirements and source code implementations 18 alejandro velasco, jairo aponte vii volume 3, number 2 may august 2022 e-issn: 2711-4627 paradigmplus e-issn: 2711-4627 volume 3, number 2 may august 2022 bogota, colombia https://journals.itiud.org/index.php/paradigmplus doi: https://doi.org/10.55969/paradigmplus editors hector florez id universidad distrital francisco josé de caldas, colombia ixent galpin id universidad de bogotá jorge tadeo lozano, colombia christian grévisse id université du luxembourg, luxembourg ivan mura id duke kunshan university, china fernando yepes-calderon id children’s hospital los angeles, united states this number and its articles are published under the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. corresponding editor: hector florez universidad distrital francisco josé de caldas, colombia haflorezf@udistrital.edu.co https://journals.itiud.org/index.php/paradigmplus https://doi.org/10.55969/paradigmplus https://orcid.org/0000-0002-5339-4459 https://orcid.org/0000-0001-7020-6328 https://orcid.org/0000-0002-9585-1160 https://orcid.org/0000-0001-5549-0324 https://orcid.org/0000-0001-9184-787x haflorezf@udistrital.edu.co journal information paradigmplus is a peer-reviewed journal published by the iti research group1 in collaboration with science based platforms for a global audience of professionals in computer science interested in disseminating computer-based research in all knowledge areas. focus and scope. paradigmplus publishes well-developed manuscripts describing theoretical, empirical, conceptual, and experimental results in all areas of computer science. the journal will address (but will not limit itself to the following areas: algorithms and analysis of algorithms, artificial intelligence, bioinformatics, business analytics, brain-computer interface, cloud computing, computer-aided design, computer architecture, computer vision, cryptography, data analysis, data mining, data science, decision systems, distributed and parallel computing, enterprise information systems applications, gamification and serious games, geoinformatics, health care information systems, human-computer interaction, interaction design and usability, it architectures, learning management systems, mobile information processing systems, natural language, robotic autonomy, software, and systems modeling, software architectures, software design engineering, and theory and formal methods. publication frequency. paradigmplus publishes one volume per year and three numbers per volume. the first number was published in april 2020. peer-review policy. paradigmplus performs a double-blind peer-review process, where the reviewers do not know the names or affiliations of the authors and the reviewer reports provided to the authors are anonymous. every submission will be reviewed at least by two experts. reviewers will also be asked to indicate how innovative and significant the research is. the editors will reach a decision based on these reports. open access policy. paradigmplus is open-access allowing free access to its content in order to foster the interchange of global knowledge. article processing charges. paradigmplus is completely free to both readers and authors. the manuscripts are submitted, evaluated, proofread, and published without any cost. creative commons licence. paradigmplus is licensed under a creative commons license attribution 4.0 international copyright terms. the corresponding author, acting on behalf of all of the authors must complete, print, hand-written sign, scan, and submit the author agreement to publish a contribution as openaccess on paradigmplus document, where the authors: • agree that the contribution is made available publicly under the creative commons license 1https://www.itiud.org/index.php?lang=en ii https://www.itiud.org/index.php?lang=en attribution 4.0 international. • grant the editors and its archiving partners the non-exclusive and irrevocable right to archive the contribution and to make it accessible (online and free of charge) for public distribution. • agree that the author names and affiliations are part of the associated metadata and may be stored. • acknowledge that the editors hold the copyright for the journal paradigmplus. • have not included any third-party material such as figures, code, data sets, and others without having properly indicated its source and credited its original authors. • warrant that the contribution does not infringe on any rights of third parties. • understand and agree that the full responsibility for the content of the contribution rests upon the authors of the contribution. • release the journals’ editors and the archiving partners of the journal paradigmplus from any liability caused by the publication or archiving of the contribution. • have read and completely understand the conditions of the creative commons license attribution 4.0 international and agree to apply this license to the contribution. anti-plagiarism policy. every submission is screened by turnitin. submissions with a high rate of coincidences are declined by the editor without review. publication ethics and malpractice statement. paradigmplus ensures that all involved actors i.e., authors, reviewers, and editors strictly follow the international rules on ethics in the publication process. thus, paradigmplus follows the ethics published by the committee on publication ethics (cope) in their cope best practice guidelines for journal editors and the international standards for editors and authors, which includes the following aspects: • general duties and responsibilities of the editorial board. having overall responsibility for the journal, the paradigmplus editorial board is committed to working together to meet the readers’ and authors’ needs, fostering the continuous improvement of the journal, ensuring the quality of published material, ensuring freedom of expression, maintaining the academic integrity of their content, and preventing commercial interests from compromising intellectual standards, posting corrections, clarifications, retractions, and apologies when necessary. • relations with readers. readers will be informed about research funders and their role in the research. • relations with authors. paradigmplus is committed to ensuring the quality of its published material, informing the goals and standards of the journal. the decision of editors to accept or decline a submission for publication is based on the relevance of the work, originality, and pertinence of the study regarding the journal’s focus and scope. paradigmplus includes a description of the process for peer evaluation of each work received and provides a guide for authors with this information. the guide is up to date and contains a link to the code of ethics. paradigmplus recognizes the right of authors to appeal to editorial decisions. editors will not change their decision in accepting or declining submission unless extraordinary circumstances or irregularities are detected. any change in the composition of the editorial board will not affect decisions already made, except in unusual cases where serious circumstances converge. iii • relations with reviewers. paradigmplus makes available to reviewers a guide to what is expected from them. during the review process, reviewers’ identity is protected at all times, ensuring anonymity. nevertheless, when a submission is accepted, reviewers’ information is included in the published article. • claims. paradigmplus is committed to answering quickly complaints, ensuring a smooth process of resolution. furthermore, if a claimant is still dissatisfied, paradigmplus considers that they have the right to raise their protests to other instances. • promoting academic integrity. paradigmplus ensures that its published material conforms to internationally-accepted ethical standards. • protection of individual data. paradigmplus guarantees the confidentiality of individuals’ information. • tracking malpractice. paradigmplus accepts the obligation to act appropriately in the event of suspected malpractice or misconduct. the editors not only reject manuscripts when there are doubts about possible misconduct but they are also considered ethically obliged to report suspected cases of misconduct. every reasonable effort is made by paradigmplus to ensure that works submitted for evaluation are rigorous in their methods and ethically appropriate. • integrity and academic rigor. whenever there is evidence that a published work contains significant misstatements, misleading, or distorted statements, it must be corrected immediately. in the event of fraudulent content being detected, the work will be removed as soon as that is known, and readers and indexing systems will be informed immediately. practices that are considered unacceptable will be reported. • conflict of interest. paradigmplus will establish the necessary mechanisms to avoid or resolve potential conflicts of interest between authors, reviewers, and the editorial board. iv editorial paradigmplus is an initiative that was born at the international conference on applied informatics (icai)2. some program committee members decided to undertake this journal to provide a new scenario for publishing high-quality articles. to achieve this, all submitted articles underwent a doubleblind peer review process, where each article was reviewed by at least two experts from the editorial team. paradigmplus is supported by the editorial team members, who are researchers that hold ph.d. degrees in computing areas. editorial team members come from the following countries: argentina, brazil, cameroon, chile, china, colombia, cyprus, ecuador, france, germany, greece, india, japan, luxembourg, morocco, netherlands, portugal, spain, switzerland, united arab emirates, united kingdom, and united states. this number contains the articles reviewed and accepted from january to april 2022. in this period, paradigmplus received 5 submissions. after a careful discussion, the editorial board decided to accept 2 articles. we thank the editorial team members for their efforts. august 2022 hector florez ixent galpin christian grévisse ivan mura fernando yepes-calderón 2https://icai.itiud.org/ v https://icai.itiud.org/ organization editor-in-chief hector florez universidad distrital francisco josé de caldas, colombia associate editors ixent galpin universidad de bogotá jorge tadeo lozano, colombia christian grévisse université du luxembourg, luxembourg ivan mura duke kunshan university, china fernando yepes-calderon children’s hospital los angeles, united states reviewers in this number saharnaz dilmaghani pricewaterhousecoopers, luxembourg ixent galpin universidad de bogotá jorge tadeo lozano, colombia olmer garcia universidad de bogotá jorge tadeo lozano, colombia christian grévisse université du luxembourg, luxembourg musonda kapatamoyo southern illinois university edwardsville, united states thomas luft j.m. voith se & co. kg, germany diego peluffo-ordóñez mohammed vi polytechnic university, morocco giovanni daian rottoli universidad tecnológica nacional, argentina fernando yepes-calderon children’s hospital los angeles, united states vi table of contents societal xr—a vision paper 1 daniel görlich predictive analysis of mental health conditions using adaboost algorithm 11 elizabeth oluyemisi ogunseye, cecilia ajowho adenusi, andrew c. nwanakwaugwu, sunday adeola ajagbe, solomon o. akinola vii volume 4, number 1 january april 2023 e-issn: 2711-4627 paradigmplus e-issn: 2711-4627 volume 4, number 1 january april 2023 bogota, colombia https://journals.itiud.org/index.php/paradigmplus doi: https://doi.org/10.55969/paradigmplus editors hector florez id universidad distrital francisco josé de caldas, colombia ixent galpin id universidad de bogotá jorge tadeo lozano, colombia christian grévisse id université du luxembourg, luxembourg ivan mura id duke kunshan university, china fernando yepes-calderon id children’s hospital los angeles, united states this number and its articles are published under the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. corresponding editor: hector florez universidad distrital francisco josé de caldas, colombia haflorezf@udistrital.edu.co https://journals.itiud.org/index.php/paradigmplus https://doi.org/10.55969/paradigmplus https://orcid.org/0000-0002-5339-4459 https://orcid.org/0000-0001-7020-6328 https://orcid.org/0000-0002-9585-1160 https://orcid.org/0000-0001-5549-0324 https://orcid.org/0000-0001-9184-787x haflorezf@udistrital.edu.co journal information paradigmplus is a peer-reviewed journal published by the iti research group1 in collaboration with science based platforms for a global audience of professionals in computer science interested in disseminating computer-based research in all knowledge areas. focus and scope. paradigmplus publishes well-developed manuscripts describing theoretical, empirical, conceptual, and experimental results in all areas of computer science. the journal will address (but will not limit itself to the following areas: algorithms and analysis of algorithms, artificial intelligence, bioinformatics, business analytics, brain-computer interface, cloud computing, computer-aided design, computer architecture, computer vision, cryptography, data analysis, data mining, data science, decision systems, distributed and parallel computing, enterprise information systems applications, gamification and serious games, geoinformatics, health care information systems, human-computer interaction, interaction design and usability, it architectures, learning management systems, mobile information processing systems, natural language, robotic autonomy, software, and systems modeling, software architectures, software design engineering, and theory and formal methods. publication frequency. paradigmplus publishes one volume per year and three numbers per volume. the first number was published in april 2020. peer-review policy. paradigmplus performs a double-blind peer-review process, where the reviewers do not know the names or affiliations of the authors and the reviewer reports provided to the authors are anonymous. every submission will be reviewed at least by two experts. reviewers will also be asked to indicate how innovative and significant the research is. the editors will reach a decision based on these reports. open access policy. paradigmplus is open-access allowing free access to its content in order to foster the interchange of global knowledge. article processing charges. paradigmplus is completely free to both readers and authors. the manuscripts are submitted, evaluated, proofread, and published without any cost. creative commons licence. paradigmplus is licensed under a creative commons license attribution 4.0 international copyright terms. the corresponding author, acting on behalf of all of the authors must complete, print, hand-written sign, scan, and submit the author agreement to publish a contribution as openaccess on paradigmplus document, where the authors: • agree that the contribution is made available publicly under the creative commons license 1https://www.itiud.org/index.php?lang=en ii https://www.itiud.org/index.php?lang=en attribution 4.0 international. • grant the editors and its archiving partners the non-exclusive and irrevocable right to archive the contribution and to make it accessible (online and free of charge) for public distribution. • agree that the author names and affiliations are part of the associated metadata and may be stored. • acknowledge that the editors hold the copyright for the journal paradigmplus. • have not included any third-party material such as figures, code, data sets, and others without having properly indicated its source and credited its original authors. • warrant that the contribution does not infringe on any rights of third parties. • understand and agree that the full responsibility for the content of the contribution rests upon the authors of the contribution. • release the journals’ editors and the archiving partners of the journal paradigmplus from any liability caused by the publication or archiving of the contribution. • have read and completely understand the conditions of the creative commons license attribution 4.0 international and agree to apply this license to the contribution. anti-plagiarism policy. every submission is screened by turnitin. submissions with a high rate of coincidences are declined by the editor without review. publication ethics and malpractice statement. paradigmplus ensures that all involved actors i.e., authors, reviewers, and editors strictly follow the international rules on ethics in the publication process. thus, paradigmplus follows the ethics published by the committee on publication ethics (cope) in their cope best practice guidelines for journal editors and the international standards for editors and authors, which includes the following aspects: • general duties and responsibilities of the editorial board. having overall responsibility for the journal, the paradigmplus editorial board is committed to working together to meet the readers’ and authors’ needs, fostering the continuous improvement of the journal, ensuring the quality of published material, ensuring freedom of expression, maintaining the academic integrity of their content, and preventing commercial interests from compromising intellectual standards, posting corrections, clarifications, retractions, and apologies when necessary. • relations with readers. readers will be informed about research funders and their role in the research. • relations with authors. paradigmplus is committed to ensuring the quality of its published material, informing the goals and standards of the journal. the decision of editors to accept or decline a submission for publication is based on the relevance of the work, originality, and pertinence of the study regarding the journal’s focus and scope. paradigmplus includes a description of the process for peer evaluation of each work received and provides a guide for authors with this information. the guide is up to date and contains a link to the code of ethics. paradigmplus recognizes the right of authors to appeal to editorial decisions. editors will not change their decision in accepting or declining submission unless extraordinary circumstances or irregularities are detected. any change in the composition of the editorial board will not affect decisions already made, except in unusual cases where serious circumstances converge. iii • relations with reviewers. paradigmplus makes available to reviewers a guide to what is expected from them. during the review process, reviewers’ identity is protected at all times, ensuring anonymity. nevertheless, when a submission is accepted, reviewers’ information is included in the published article. • claims. paradigmplus is committed to answering quickly complaints, ensuring a smooth process of resolution. furthermore, if a claimant is still dissatisfied, paradigmplus considers that they have the right to raise their protests to other instances. • promoting academic integrity. paradigmplus ensures that its published material conforms to internationally-accepted ethical standards. • protection of individual data. paradigmplus guarantees the confidentiality of individuals’ information. • tracking malpractice. paradigmplus accepts the obligation to act appropriately in the event of suspected malpractice or misconduct. the editors not only reject manuscripts when there are doubts about possible misconduct but they are also considered ethically obliged to report suspected cases of misconduct. every reasonable effort is made by paradigmplus to ensure that works submitted for evaluation are rigorous in their methods and ethically appropriate. • integrity and academic rigor. whenever there is evidence that a published work contains significant misstatements, misleading, or distorted statements, it must be corrected immediately. in the event of fraudulent content being detected, the work will be removed as soon as that is known, and readers and indexing systems will be informed immediately. practices that are considered unacceptable will be reported. • conflict of interest. paradigmplus will establish the necessary mechanisms to avoid or resolve potential conflicts of interest between authors, reviewers, and the editorial board. iv editorial paradigmplus is an initiative that was born at the international conference on applied informatics (icai)2. some program committee members decided to undertake this journal to provide a new scenario for publishing high-quality articles. to achieve this, all submitted articles underwent a doubleblind peer review process, where each article was reviewed by at least two experts from the editorial team. paradigmplus is supported by the editorial team members, who are researchers that hold ph.d. degrees in computing areas. editorial team members come from the following countries: argentina, brazil, cameroon, chile, china, colombia, cyprus, ecuador, france, germany, greece, india, japan, luxembourg, morocco, netherlands, nigeria, norway, portugal, south africa, spain, switzerland, united arab emirates, united kingdom, and united states. this number contains the articles reviewed and accepted from january to april 2023. in this period, paradigmplus received 5 submissions. after a careful discussion, the editorial board decided to accept 2 articles. we thank the editorial team members for their efforts. april 2023 hector florez ixent galpin christian grévisse ivan mura fernando yepes-calderón 2https://icai.itiud.org/ v https://icai.itiud.org/ organization editor-in-chief hector florez universidad distrital francisco josé de caldas, colombia associate editors ixent galpin universidad de bogotá jorge tadeo lozano, colombia christian grévisse université du luxembourg, luxembourg ivan mura duke kunshan university, china fernando yepes-calderon children’s hospital los angeles, united states reviewers in this number luis de-la-fuente-valentin universidad internacional de la rioja, spain cesar diaz ocox ai, colombia olmer garcia universidad de bogotá jorge tadeo lozano, colombia musonda kapatamoyo southern illinois university edwardsville, united states diego peluffo-ordóñez mohammed vi polytechnic university, morocco florencia pollo-cattaneo universidad tecnológica nacional, argentina luis rabelo university of central florida, united states sweta singh savitribai phule pune university, india fernando yepes-calderon children’s hospital los angeles, united states vi table of contents continuous eye disease severity evaluation system using siamese neural networks 1 muyideen abdulraheem, idowu d. oladipo, sunday adeola ajagbe, ghaniyyat b. balogun, mulikat bola akanbi, nissi o. emma-adamah development of an improved convolutional neural network for an automated face based university attendance system 18 olufemi s. ojo, mayowa o. oyediran, babatunde j. bamgbade, abidemi emmanuel adeniyi, godwin nse ebong, sunday a. ajagbe vii 0110011011110110101111110100000100011100010100011111001100101110101000110001110000001100100001110101001110011 1100100011010100100011111010111100000010011010100101000000000101001000001000010001011001100101000000110010010 1111100110000100100110101010100010101001110101110010001011101100101001000001001111100011101001010010011011011 0110111001110110100101111110010100010100010001100010000110010100010000001001001111101111000011101101000100111 0000111110011011100000110100111111110000110011011100110001001110011001101110101101101011001111111100010011001 0000010011011111000010001111101110011101110101101010110011100101001011011000001001100100010110011000011111010 0101000100011010001011010010000011110100000011110110101000101111001000011101001100011011111101111100111111010 1010010001011101110111101101110010001100010101100010110010111001101100111111010111100100001011111011011010010 0011000011101100101010000011111110000101110000010010111100111011100000001111100001111001101000110100101000000 0000110111100000000110111010010000010110001101001011110100001000110110010101010001111110100100000000110101110 0100001110010001000011101101101100000101101111000000110000110101011010000111001100110001011110001010101101000 0101001111000111011010011101001101001110100010010111000001010110100001111001010000001101110110111101001010011 0111000100110000100100100001110000110010000001000010110111000010010101011111100110010101110001100010010001000 0111001111011100000010101101101101010100101111010101011010010011111111110011001000011010011011001100110011100 0000100111001000011010010111011100111000000101111111001100111111111111011100010111111100000100110011101100001 1011101001100010010000000011011110101101111101010011001111000110101101000001111011010000111100100011000110111 1101110010100011100000110011001100100000101011100101000010011101010111100100010001111001101001010001110100011 1100010110111100000101110110101100001010110010101110111010011101011010010110101010001100101101110101111000000 0100010000010000101001111101101100001101110001010011001111000000001010010110010110101001010101010001110000101 0000010110111011010101110110001001010011101010101101111110011111000101001110010001100110001101010000000011010 1001011011110001111111000100011011110110011000011010000101001100011100100110110000011110111010001000000111010 0101100110111000101100100110000011011001011011110010001000110100110010101000001110111111011110101001010101111 1101011001000110000110011100110011010010000011111100011110011010101000000010011001000110101010110101000000101 volume 1, number 3 september december 2020 e-issn: 2711-4627 paradigmplus e-issn: 2711-4627 volume 1, number 3 september december 2020 bogota, colombia https://journals.itiud.org/index.php/paradigmplus doi: https://doi.org/10.55969/paradigmplus editors hector florez id universidad distrital francisco josé de caldas bogota, colombia ixent galpin id universidad de bogotá jorge tadeo lozano bogota, colombia christian grévisse id université du luxembourg esch-sur-alzette, luxembourg fernando yepes-calderon id children’s hospital los angeles los angeles, united states this number and its articles are published under the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. corresponding editor: hector florez universidad distrital francisco josé de caldas bogota, colombia haflorezf@udistrital.edu.co https://journals.itiud.org/index.php/paradigmplus https://doi.org/10.55969/paradigmplus https://orcid.org/0000-0002-5339-4459 https://orcid.org/0000-0001-7020-6328 https://orcid.org/0000-0002-9585-1160 https://orcid.org/0000-0001-9184-787x haflorezf@udistrital.edu.co journal information paradigmplus is a peer-reviewed journal published by the iti research group1 in collaboration with science based platforms for a global audience of professionals in computer science interested in disseminating computer-based research in all knowledge areas. focus and scope. paradigmplus publishes well-developed manuscripts describing theoretical, empirical, conceptual, and experimental results in all areas of computer science. the journal will address (but will not limit itself to the following areas: algorithms and analysis of algorithms, artificial intelligence, bioinformatics, business analytics, brain-computer interface, cloud computing, computer-aided design, computer architecture, computer vision, cryptography, data analysis, data mining, data science, decision systems, distributed and parallel computing, enterprise information systems applications, gamification and serious games, geoinformatics, health care information systems, human-computer interaction, interaction design and usability, it architectures, learning management systems, mobile information processing systems, natural language, robotic autonomy, software and systems modeling, software architectures, software design engineering, and theory and formal methods. publication frequency. paradigmplus publishes one volume per year and three numbers per volume. the first number was published in april 2020. peer-review policy. paradigmplus performs a double-blind peer-review process, where the reviewers do not know the names or affiliations of the authors and the reviewer reports provided to the authors are anonymous. every submission will be reviewed at least by two experts. reviewers will also be asked to indicate how innovative and significant the research is. the editors will reach a decision based on these reports. when a submission is accepted, the editor will include the names and affiliation of the reviewers in alphabetical order by last name. open access policy. paradigmplus is open-access allowing free access to its content in order to foster the interchange of global knowledge. article processing charges. paradigmplus is completely free to both readers and authors. the manuscripts are submitted, evaluated, proofread, and published without any cost. creative commons licence. paradigmplus is licensed under a creative commons license attribution 4.0 international copyright terms. the corresponding author, acting on behalf of all of the authors must complete, print, hand-written sign, scan, and submit the author agreement to publish a contribution as openaccess on paradigmplus document, where authors: 1https://www.itiud.org/index.php?lang=en ii https://www.itiud.org/index.php?lang=en • agree that the contribution is made available publicly under the creative commons license attribution 4.0 international. • grant the editors and its archiving partners the non-exclusive and irrevocable right to archive the contribution and to make it accessible (online and free of charge) for public distribution. • agree that the author names and affiliations are part of the associated metadata and may be stored. • acknowledge that the editors hold the copyright for the journal paradigmplus. • have not included any third-party material such as figures, code, data sets, and others without having properly indicated its source and credited its original authors. • warrant that the contribution does not infringe on any rights of third parties. • understand and agree that the full responsibility for the content of the contribution rests upon the authors of the contribution. • release the journals’ editors and the archiving partners of the journal paradigmplus from any liability caused by the publication or archiving of the contribution. • have read and completely understand the conditions of the creative commons license attribution 4.0 international and agree to apply this license to the contribution. anti-plagiarism policy. every submission is screened by turnitin. submissions with a high rate of coincidences are declined by the editor without review. publication ethics and malpractice statement. paradigmplus ensures that all involved actors i.e., authors, reviewers, and editors strictly follow the international rules on ethics in the publication process. thus, paradigmplus follows the ethics published by the committee on publication ethics (cope) in their cope best practice guidelines for journal editors and the international standards for editors and authors, which includes the following aspects: • general duties and responsibilities of the editorial board. having overall responsibility for the journal, the paradigmplus editorial board are committed to: working together to meet the readers’ and authors’ needs, fostering the continuous improvement of the journal, ensuring the quality of published material, ensuring freedom of expression, maintaining the academic integrity of their content, and preventing commercial interests from compromising intellectual standards, posting corrections, clarifications, retractions, and apologies when necessary. • relations with readers. readers will be informed about research funders and their role in the research. • relations with authors. paradigmplus is committed to ensuring the quality of its published material, informing the goals and standards of the journal. the decision of editors to accept or decline a submission for publication is based on the relevance of the work, originality, and pertinence of the study regarding the journal’s focus and scope. paradigmplus includes a description of the process for peer evaluation of each work received and provides a guide for authors with this information. the guide is up to date and contains a link to the code of ethics. paradigmplus recognizes the right of authors to appeal to editorial decisions. editors will not change their decision in accepting or declining submission unless extraordinary circumstances or irregularities are detected. any change in the composition of the editorial board will not affect decisions already made, except in unusual cases where serious circumstances converge. iii • relations with reviewers. paradigmplus makes available to reviewers a guide to what is expected from them. during the review process, reviewers’ identity is protected at all times, ensuring anonymity. nevertheless, when a submission is accepted, reviewers’ information is included in the published article. • claims. paradigmplus is committed to answering quickly to complaints, ensuring a smooth process of resolution. furthermore, if a claimant is still dissatisfied, paradigmplus considers that they have the right to raise their protests to other instances. • promoting academic integrity. paradigmplus ensures that its published material conforms to internationally-accepted ethical standards. • protection of individual data. paradigmplus guarantees the confidentiality of individuals’ information. • tracking malpractice. paradigmplus accepts the obligation to act appropriately in the event of suspected malpractice or misconduct. the editors not only reject manuscripts when there are doubts about possible misconduct, but they are also considered ethically obliged to report suspected cases of misconduct. every reasonable effort is made by paradigmplus to ensure that works submitted for evaluation are rigorous in their methods and ethically appropriate. • integrity and academic rigor. whenever there is evidence that a published work contains significant misstatements, misleading, or distorted statements, it must be corrected immediately. in the event of fraudulent content being detected, the work will be removed as soon as that is known, and readers and indexing systems will be informed immediately. practices that are considered unacceptable will be reported. • conflict of interest. paradigmplus will establish the necessary mechanisms to avoid or resolve potential conflicts of interest between authors, reviewers, and the editorial board. iv editorial paradigmplus is an initiative that was born in the international conference on applied informatics (icai)2. some members of the program committee that participated in icai 2018 and icai 2019 decided to undertake this journal in order to provide a new scenario for publishing high quality articles. to achieve this, all submitted articles underwent a double blind peer review process, where each article was reviewed by at least three experts from the editorial team. paradigmplus is supported by the editorial team members, who are researchers that hold ph.d. degrees in computing areas. editorial team members come from the following countries: argentina, brazil, chile, china, colombia, cyprus, ecuador, germany, greece, india, japan, luxembourg, morocco, netherlands, spain, switzerland, united arab emirates, united kingdom, and united states. this number contains the articles reviewed and accepted from september to december 2020. in this period, paradigmplus received 4 submissions. after a careful discussion, the editorial board decided to accept the articles: a language and methodology based on scenarios, grammars and views, for administrative business processes modelling, whose authors are from cameroon and france and assessing employee satisfaction in the context of covid-19 pandemic, whose authors are from portugal. we thank the editorial team members for their efforts. december 2020 hector florez ixent galpin christian grévisse fernando yepes-calderón 2https://icai.itiud.org/ v https://icai.itiud.org/ organization editor-in-chief hector florez universidad distrital francisco josé de caldas, colombia associate editors ixent galpin universidad de bogotá jorge tadeo lozano, colombia christian grévisse université du luxembourg, luxembourg fernando yepes-calderon children’s hospital los angeles, united states reviewers in this number raphael gomes instituto federal de goiás, brazil jens gulden universiteit utrecht, netherlands tong li beijing university of technology, china diego peluffo-ordóñez universidad yachay tech, ecuador ben roelens open universiteit, netherlands fernando yepes-calderon children’s hospital los angeles, united states vi table of contents a language and methodology based on scenarios, grammars and views, for administrative business processes modelling 1 milliam maxime zekeng ndadji, maurice tchoupé tchendji, clémentin tayou djamegni, didier parigot assessing employee satisfaction in the context of covid-19 pandemic 23 ana fernandes, rui lima, margarida figueiredo, jorge ribeiro, josé neves, henrique vicente vii volume 3, number 3 september december 2022 e-issn: 2711-4627 paradigmplus e-issn: 2711-4627 volume 3, number 3 september december 2022 bogota, colombia https://journals.itiud.org/index.php/paradigmplus doi: https://doi.org/10.55969/paradigmplus editors hector florez id universidad distrital francisco josé de caldas, colombia ixent galpin id universidad de bogotá jorge tadeo lozano, colombia christian grévisse id université du luxembourg, luxembourg ivan mura id duke kunshan university, china fernando yepes-calderon id children’s hospital los angeles, united states this number and its articles are published under the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. corresponding editor: hector florez universidad distrital francisco josé de caldas, colombia haflorezf@udistrital.edu.co https://journals.itiud.org/index.php/paradigmplus https://doi.org/10.55969/paradigmplus https://orcid.org/0000-0002-5339-4459 https://orcid.org/0000-0001-7020-6328 https://orcid.org/0000-0002-9585-1160 https://orcid.org/0000-0001-5549-0324 https://orcid.org/0000-0001-9184-787x haflorezf@udistrital.edu.co journal information paradigmplus is a peer-reviewed journal published by the iti research group1 in collaboration with science based platforms for a global audience of professionals in computer science interested in disseminating computer-based research in all knowledge areas. focus and scope. paradigmplus publishes well-developed manuscripts describing theoretical, empirical, conceptual, and experimental results in all areas of computer science. the journal will address (but will not limit itself to the following areas: algorithms and analysis of algorithms, artificial intelligence, bioinformatics, business analytics, brain-computer interface, cloud computing, computer-aided design, computer architecture, computer vision, cryptography, data analysis, data mining, data science, decision systems, distributed and parallel computing, enterprise information systems applications, gamification and serious games, geoinformatics, health care information systems, human-computer interaction, interaction design and usability, it architectures, learning management systems, mobile information processing systems, natural language, robotic autonomy, software, and systems modeling, software architectures, software design engineering, and theory and formal methods. publication frequency. paradigmplus publishes one volume per year and three numbers per volume. the first number was published in april 2020. peer-review policy. paradigmplus performs a double-blind peer-review process, where the reviewers do not know the names or affiliations of the authors and the reviewer reports provided to the authors are anonymous. every submission will be reviewed at least by two experts. reviewers will also be asked to indicate how innovative and significant the research is. the editors will reach a decision based on these reports. open access policy. paradigmplus is open-access allowing free access to its content in order to foster the interchange of global knowledge. article processing charges. paradigmplus is completely free to both readers and authors. the manuscripts are submitted, evaluated, proofread, and published without any cost. creative commons licence. paradigmplus is licensed under a creative commons license attribution 4.0 international copyright terms. the corresponding author, acting on behalf of all of the authors must complete, print, hand-written sign, scan, and submit the author agreement to publish a contribution as openaccess on paradigmplus document, where the authors: • agree that the contribution is made available publicly under the creative commons license 1https://www.itiud.org/index.php?lang=en ii https://www.itiud.org/index.php?lang=en attribution 4.0 international. • grant the editors and its archiving partners the non-exclusive and irrevocable right to archive the contribution and to make it accessible (online and free of charge) for public distribution. • agree that the author names and affiliations are part of the associated metadata and may be stored. • acknowledge that the editors hold the copyright for the journal paradigmplus. • have not included any third-party material such as figures, code, data sets, and others without having properly indicated its source and credited its original authors. • warrant that the contribution does not infringe on any rights of third parties. • understand and agree that the full responsibility for the content of the contribution rests upon the authors of the contribution. • release the journals’ editors and the archiving partners of the journal paradigmplus from any liability caused by the publication or archiving of the contribution. • have read and completely understand the conditions of the creative commons license attribution 4.0 international and agree to apply this license to the contribution. anti-plagiarism policy. every submission is screened by turnitin. submissions with a high rate of coincidences are declined by the editor without review. publication ethics and malpractice statement. paradigmplus ensures that all involved actors i.e., authors, reviewers, and editors strictly follow the international rules on ethics in the publication process. thus, paradigmplus follows the ethics published by the committee on publication ethics (cope) in their cope best practice guidelines for journal editors and the international standards for editors and authors, which includes the following aspects: • general duties and responsibilities of the editorial board. having overall responsibility for the journal, the paradigmplus editorial board is committed to working together to meet the readers’ and authors’ needs, fostering the continuous improvement of the journal, ensuring the quality of published material, ensuring freedom of expression, maintaining the academic integrity of their content, and preventing commercial interests from compromising intellectual standards, posting corrections, clarifications, retractions, and apologies when necessary. • relations with readers. readers will be informed about research funders and their role in the research. • relations with authors. paradigmplus is committed to ensuring the quality of its published material, informing the goals and standards of the journal. the decision of editors to accept or decline a submission for publication is based on the relevance of the work, originality, and pertinence of the study regarding the journal’s focus and scope. paradigmplus includes a description of the process for peer evaluation of each work received and provides a guide for authors with this information. the guide is up to date and contains a link to the code of ethics. paradigmplus recognizes the right of authors to appeal to editorial decisions. editors will not change their decision in accepting or declining submission unless extraordinary circumstances or irregularities are detected. any change in the composition of the editorial board will not affect decisions already made, except in unusual cases where serious circumstances converge. iii • relations with reviewers. paradigmplus makes available to reviewers a guide to what is expected from them. during the review process, reviewers’ identity is protected at all times, ensuring anonymity. nevertheless, when a submission is accepted, reviewers’ information is included in the published article. • claims. paradigmplus is committed to answering quickly complaints, ensuring a smooth process of resolution. furthermore, if a claimant is still dissatisfied, paradigmplus considers that they have the right to raise their protests to other instances. • promoting academic integrity. paradigmplus ensures that its published material conforms to internationally-accepted ethical standards. • protection of individual data. paradigmplus guarantees the confidentiality of individuals’ information. • tracking malpractice. paradigmplus accepts the obligation to act appropriately in the event of suspected malpractice or misconduct. the editors not only reject manuscripts when there are doubts about possible misconduct but they are also considered ethically obliged to report suspected cases of misconduct. every reasonable effort is made by paradigmplus to ensure that works submitted for evaluation are rigorous in their methods and ethically appropriate. • integrity and academic rigor. whenever there is evidence that a published work contains significant misstatements, misleading, or distorted statements, it must be corrected immediately. in the event of fraudulent content being detected, the work will be removed as soon as that is known, and readers and indexing systems will be informed immediately. practices that are considered unacceptable will be reported. • conflict of interest. paradigmplus will establish the necessary mechanisms to avoid or resolve potential conflicts of interest between authors, reviewers, and the editorial board. iv editorial paradigmplus is an initiative that was born at the international conference on applied informatics (icai)2. some program committee members decided to undertake this journal to provide a new scenario for publishing high-quality articles. to achieve this, all submitted articles underwent a doubleblind peer review process, where each article was reviewed by at least two experts from the editorial team. paradigmplus is supported by the editorial team members, who are researchers that hold ph.d. degrees in computing areas. editorial team members come from the following countries: argentina, brazil, cameroon, chile, china, colombia, cyprus, ecuador, france, germany, greece, india, japan, luxembourg, morocco, netherlands, portugal, spain, switzerland, united arab emirates, united kingdom, and united states. this number contains the articles reviewed and accepted from january to april 2022. in this period, paradigmplus received 6 submissions. after a careful discussion, the editorial board decided to accept 2 articles. we thank the editorial team members for their efforts. december 2022 hector florez ixent galpin christian grévisse ivan mura fernando yepes-calderón 2https://icai.itiud.org/ v https://icai.itiud.org/ organization editor-in-chief hector florez universidad distrital francisco josé de caldas, colombia associate editors ixent galpin universidad de bogotá jorge tadeo lozano, colombia christian grévisse université du luxembourg, luxembourg ivan mura duke kunshan university, china fernando yepes-calderon children’s hospital los angeles, united states reviewers in this number hüseyin bicen yakin dogu üniversitesi, cyprus cesar diaz ocox ai, colombia ixent galpin universidad de bogotá jorge tadeo lozano, colombia olmer garcia universidad de bogotá jorge tadeo lozano, colombia musonda kapatamoyo southern illinois university edwardsville, united states thomas luft j.m. voith se & co. kg, germany ivan mura, duke kunshan university, china florencia pollo-cattaneo universidad tecnológica nacional, argentina luis rabelo university of central florida, united states vi table of contents comparison of the performance of machine learning techniques in the prediction of employee 1 jide kehinde adeniyi, abidemi emmanuel adeniyi, yetunde josephine oguns, gabriel olumide egbedokun, kehinde douglas ajagbe, princewill chima obuzor, sunday adeola ajagbe hybrid approach for phishing website detection using classification algorithms 16 mukta mithra raj, j. angel arul jothi vii vol. 4, no. 1, pp 18 28, 2023 doi: 10.55969/paradigmplus.v4n1a2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . development of an improved convolutional neural network for an automated face-based university attendance system olufemi s. ojo1, mayowa o. oyediran1, babatunde j. bamgbade id 2, abidemi emmanuel adeniyi id 3, godwin nse ebong id 4, and sunday adeola ajagbe id 5,� 1ajayi crowther university, oyo, nigeria {os.ojo, mo.oyediran}@acu.edu.ng 2federal college of forestry, ibadan, nigeria bamgbade.bj@frin.gov.ng 3precious cornerstone university, ibadan, nigeria emmanueladeniyi@pcu.edu.ng 4the university of salford, manchester, united kingdom g.n.ebong@edu.salford.ac.uk 5first technical university ibadan, idaban, nigeria sunday.ajagbe@tech-u.edu.ng abstract because of the flaws of the present university attendance system, which has always been time intensive, not accurate, and a hard process to follow. it, therefore, becomes imperative to eradicate or minimize the deficiencies identified in the archaic method. the identification of human face systems has evolved into a significant element in autonomous attendance-taking systems due to their ease of adoption and dependable and polite engagement. face recognition technology has drastically altered the field of convolution neural networks (cnn) however it has challenges of high computing costs for analyzing information and determining the best specifications (design) for each problem. thus, this study aims to enhance cnn’s performance using genetic algorithm (ga) for an automated face-based university attendance system. the improved face recognition accuracy with cnn-ga got 96.49% while the face recognition accuracy with cnn got 92.54%. keywords: attendance system ⋅ cnn ⋅ data mining ⋅ face recognition ⋅ genetic algorithm received: 10 march 2023 ⋅ accepted: 20 april 2023 ⋅ published: 27 april 2023. 1 introduction collecting attendance records in classroom settings is one of the most regularly utilized strategies in many colleges to enhance students’ academic performance. there is a considerable correlation between student presence and educational success; low participation is typically associated with poor accomplishment [1]. an appropriate strategy for validating and monitoring attendance data is essential to ensure the accuracy of a student’s attendance record. the typical method of collecting attendance is to distribute an attendance sheet around the classroom and have pupils sign their names. this article is distributed under the terms of the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. https://doi.org/10.55969/paradigmplus.v4n1a2 https://orcid.org/0000-0002-3222-7002 https://orcid.org/0000-0002-2728-0116 https://orcid.org/0000-0002-9899-8838 https://orcid.org/0000-0002-7010-5540 development of an improved convolutional neural network 19 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . another frequent method is headcount, in which the lecturer registers presence by calling each student’s name. one advantage of these manual attendance-taking systems is that no specific setting or technology is required. these manual procedures, however, have clear drawbacks [2, 3]. the traditional student attendance system has been known to be a time-consuming, not accurate, and hard process to follow. it, therefore, becomes imperative to eradicate or minimize the deficiencies identified in the archaic method. these tactics not only squander important class time but also risk incorporating impostors. also, the format of class attendance numbers is difficult to handle and easily lost if not attentive. recently, a variety of facial recognition-based algorithms for managing attendance automatically have been created, including local, and hybrid methods that describe a face image using just a few of its properties or the whole thing [4, 5, 6, 7]. in the realm of computer vision, face recognition is one of the most extensively used biometric identification technologies [8]. face recognition-based attendance systems typically comprise picture collecting, dataset creation, face detection, and face recognition. a face, unlike a fingerprint, may be easily identified by a human. human face recognition systems have become a significant element in automated participation systems due to their ease of acquisition and dependable and polite interactions [9, 10, 11]. convolution neural network (cnn) has dramatically altered the area of face recognition since it can autonomously locate the features required for classification during the training phase without the requirement for feature extraction stages [12]. nevertheless, while building cnn architectures, various difficulties arise, such as high computing costs for input processing and determining the best cnn features (model) for each task [13, 14, 15, 16, 17]. thus this study aims to enhance cnn’s performance using genetic algorithm (ga) by reducing the computational expenses for data processing and determining the best cnn features (structure) for a face-based attendance system. the purpose of this research is to solve the problems that traditional university attendance systems encounter. traditional attendance methods are time-consuming and prone to mistakes, resulting in erroneous attendance records. moreover, manual attendance methods are unsuitable for big classrooms, and it was difficult to keep track of students who may attempt to trick the system via proxy attendance or faking attendance records. face recognition technology-based automatic attendance systems can circumvent these restrictions. a system like this may record attendance reliably and rapidly, minimize fraud, and remove the need for human data entry. the accuracy of facial recognition systems, on the other hand, was highly reliant on the quality of the input data and the algorithm employed for recognition. as a result, an upgraded cnn can assist increase the accuracy and efficiency of the attendance monitoring system. the improved cnn may be used to overcome typical problems in face recognition systems such as occlusion, lighting fluctuations, and position variations. this technique may be used with other technologies, such as machine learning, to enhance recognition accuracy and decrease false positives. as a result, this has the potential to deliver an efficient, precise, and dependable attendance monitoring system that may save time, eliminate mistakes, and promote student responsibility. 2 related work pandiselvi et al., [7] introduced an rfid-based smart class attendance monitoring system with human identification for absentees. the study is based on a small and dependable classroom attendance system employing an rfid card and facial recognition technology. it just utilized matlab progress to validate each student’s face. then, in the task, an individual buzzer is employed. it identifies proxy attendance and stores it in a log file. the task then observes the absentees list and sends the message to the appropriate or specified number. rfid and facial verification are used to utilize and take attendance in this system, which has a 98% efficiency. ikhar et al., [10] illustrate how an automated attendance system works in a classroom setting. there are various ways to face recognition, the most well-known of which is principle component paradigmplus (2023) 4:1 20 olufemi s. ojo et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . analysis (pca). it is suggested to use principle component analysis to create an automated attendance system that utilizes facial recognition. recognition of facial images eigenface is a method for reducing face dimensional space by utilizing pca for facial characteristics. the system is made up of a dataset including a set of face patterns for each individual. this unique technology is sufficiently safe, dependable, and widely available for usage in a variety of applications such as safety, industrial, education, facial detection, and so on. pei et al., [3] proposed a deep learning (dl) face recognition using data augmentation based on orthogonal experiments. the data augmentation methodology was introduced as a viable solution to overcome the issue of over-fitting cnn architectures due to inadequate samples. the main idea behind data augmentation is to produce virtual samples in order to enhance the size of the training dataset while decreasing over-fitting. to expand the training samples, geometric transformation changes in picture brightness, and operations employing various filter techniques are used in this research. moreover, they use the orthogonal experimental approach to examine the influence of the aforementioned processing techniques on the accuracy of facial recognition software. the original test images are then supplemented with the best data augmentation methodology based on the orthogonal experiment results. our class attendance-taking approach with data augmentation yields an accuracy of 86.3%, with more data gathered over a term improving the accuracy to 98.1% [3]. amit et al., [11] proposed an automated attendance system using facial recognition. this study uses the face recognition technique using employee records for marking attendance. in the proposed system application is active during the working hours of the organization. the camera of the system (computer application) will be fitted at the entrance as such to scan the faces of anyone who enters the office, the application captures the image and sends it to the processing side. the processing part of the application recognizes the face of the employee. finally, the application marks the employee if he/she is present. if an employee is not recognized by the application i.e., he/she is not present, he/she is marked as absent for the day. the proposed system requires less hardware support also the processing time is also less in comparison to other conventional systems of signing papers, radio frequency identification [rfid], and biometrics which proves this system to be more efficient for organizations to use in real-time application [8]. baheti et. al., [18] developed an automatic attendance system using the dl-based cnn technique for face recognition. they began by photographing pupils in a classroom and then used the opencv module to recognize and frame the faces in the picture. the frames are then enhanced using an image-improving technique in the next phase. in the last step of deployment, they develop a cnn to train these face photos and match them with the student information stored in the database, updating the students’ participation level. ahmed et. al., [19] created a real-time attendance system for pupils for use in small-scale contexts like classrooms. three separate strategies were employed to do this, including facial recognition, identity, and numbering. the histogram of oriented gradients technique is used to recognize faces. a cnn was utilized to extract facial characteristics for face recognition. to recognize individual faces in the picture, a support vector machine was utilized. lastly, during the counting stage, a haar cascade classifier was employed to count the number of faces in the picture. as a consequence, the created system identified faces in the collected image with a classification accuracy of 99.75%. the suggested method is an ideal answer for replacing the previous manual technique with an autonomous one. joseph, et al., [20] introduced a simultaneous facial detection monitoring system based on a field of intelligent machines known as cnn and opencv. the research findings demonstrated the efficiency of the suggested approach, with classification results of around 98%, a precision of 96%, and a recall of 0.96. as a result, the suggested approach is a viable face recognition system. most face recognition technologies have difficulties detecting faces in settings such as sound, position, facial expression, lighting, blockage, and low-performance correctness. filippidou et al., [21] concentrated on the example of a single sample face identification problem, with the goal of developing a real-time visual-based awareness solution. five well-known pre-trained cnns were tested in this regard. the study findings indicate that densenet121 was the best choice paradigmplus (2023) 4:1 development of an improved convolutional neural network 21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . for interacting with active learning strategies (up to 99% top-l accuracy), as well as the best and most powerful concept for interacting with the single data point per individual challenge, which is connected to using deep cnns on a limited data and, more explicitly, the mono sample per individual facial recognition software task. sawhney et. al., [22] offered a methodology for building an autonomous system to track attendance for pupils taking a class utilizing eigenface values, pca, and cnn. furthermore, the association of identified faces should be possible by matching them to the dataset held student photographs. this strategy will be an effective method for managing student attendance and documentation. dang [23] research employed an upgraded facenet model architecture based on a mobilenetv2 backbone with an ssd component. deep-wise segmented convolution was used in the enhanced framework to minimize model size and computing complexity while maintaining excellent accuracy and processing performance. to tackle the challenge of detecting a person entering and departing an area and merging highly mobile resources with limitations on sophisticated mobile devices. the technique particularly produced good outcomes in practice, with more than 95% correctness on a small sample of actual face photos. according to the various reviewed studies, it was discovered that the accuracy achieved by those studies for attendance systems was insufficient for a reliable system in real-time student identification. table 1: summary of literature review author method result limitations [7] rfid-based smart class attendance the study achieved an efficiency of 98% small sample size, absentee identification limitation with limited validation. [10] eigenface method with pca the system is secure enough, reliable, and available for use for various applications. lack of robust tracking system. [3] dl using data augmentation based on orthogonal experiments the study achieve an accuracy of 86.3% with data augmentation overfitting of training dataset [11] rfid, biometrics the study achieved lesser processing time to capture the attendance. human error in fitting the camera to capture faces [18] dl cnns the train facial images are compared with the database image for capturing attendance the capturing of attendance was not real-time [19] cnn, svm, haar. the system automatically captures student attendance and obtains an accuracy of 99.75%. limited dataset used for the classification [20] cnn, opencv the results showed a precision of 96% and recall of 0.96 difficulties in detecting faces [21] cnn the study indicate that densenet121 performance better for attendance capturing limited data with single testing sample [22] eigenface value, pca, cnn it captures attendance effectively matching problem [23] facenet on mobilenetv2 the study achieved an accuracy of 95% complexity problem paradigmplus (2023) 4:1 22 olufemi s. ojo et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . therefore, it was important to solve the prevalent problem with recognition accuracy. the research presented a solution that could handle the problems mentioned in the existing study. the research used cnn deep features and svm for real-time student face identifications to automatically mark attendance, saving time and reliability. the summary of the review of the existing works is presented in table 1. 3 methodology 3.1 data acquisition face images were acquired via digital camera from students of ajayi university, oyo, nigeria. a digital camera with at least 5 megapixels and above which makes a sharp 8-by-10-inch print was used. the captured image size of 1600 by 1200 resolution was downsized to 128 by 128-pixel resolution. sixty percent of the images were used for training and forty percent were used for testing the system using the random sampling cross-validation method. 3.2 data pre-processing an image is made up of a collection of square pixels (picture components) arranged in columns and rows. this was done to remove noise from the face images. 3.2.1 conversion to grayscale an image is made up of a collection of square pixels (picture components) arranged in columns and rows. each picture element in a (8-bit) grayscale image has a corresponding intensity value that falls between 0 and 255. the intensity of the pixel is represented by these values, where 255 denotes white pixels and 0 denotes black pixels. each character’s sample picture size varies, however for the sake of this study, a fixed size of 28 by 28 was used. 3.2.2 image normalization the histogram equalization was applied as image normalization in which the intensity of the image was distributed using a cumulative distribution function. the intensity levels of the photos were distributed using the cumulative distribution function. the mapping between the input picture grayscale values and the transformed image grayscale values was determined by the cumulative distribution function. histogram equalization was used for enhancement contrast. 3.2.3 thinning the thinning algorithm is a morphological technique that is applied to binary images to remove certain foreground pixels. 3.2.4 data augmentation technique for data augmentation such are image scaling, image rotation, and image shearing was employed in this study. 3.3 feature extraction and classification phase using deep learning approach with no requirement for manual feature selection, dl algorithms may be able to learn tasks directly from data. dl is a method for comprehending data, such as photographs, by learning several levels of representation and abstraction. end-to-end learning is used in dl for feature extraction and paradigmplus (2023) 4:1 development of an improved convolutional neural network 23 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . categorization. after obtaining a preprocessed face image after extracting features for each modality, classification was performed using a dl method that combines an optimization tool ga and a cnn. in this study, convolutional layers and subsampling max-pooling were combined to build the proposed cnn. two classification layers that were entirely connected make up the upper layers of the planned cnn. the softmax classifiers then generated a likelihood function across the n class labels using the outputs of the last fully linked layer. 3.4 training and classification the datasets were used to generate sets for training and testing. the test set was used to monitor the network’s generalization capacity during the learning phase and to record the weights configurations that were used to perform best with the least amount of validation error. the procedural steps followed to achieve the training and classification of modalities are as follows: – step 1: generate random population of n, weight space ω = [ω1, ω2, . . . , ωn], set parameter crossover probability pc, mutation probability pm. – step 2: forward pass: output of neuron of row k, column y in the lth convolution layer and kth feature pattern among them, f is the number of convolution cores in a feature pattern, the output of neuron of row x, column y in the lth sub-sample layer and kth feature pattern, the output of the jth neuron in lth hidden layer h, among them, s is the number of feature patterns in sample layer, output of the ith neuron lth output layer f . – step 3: backpropagation: output deviation of the kth neuron in output layer o. – step 4: input deviation of the kth neuron in the output layer. – step 5: weight and bias variation of kth neuron in the output o. – step 6: output bias of kth neuron in hidden layer h. – step 7: input bias of kth neuron in hidden layer h. – step 8: weight and bias variation in row x, column y in the mth feature pattern, a former layer in front of k neurons in hidden layer h. – step 9: output bias of row x, column y in mth feature pattern, sub-sample layer s. – step 10: input bias of row x, column y in mth feature pattern, sub-sample layer s. – step 11: weight and bias variation of row x, column y in mth feature pattern, sub-sample layer s. – step 12: output bias of row x, column y in kth feature pattern, convolution layer c. – step 13: input bias of row x, column y in kth feature patter, convolution layer c. – step 14: evaluate objective function based on initial optimal weight features. – step 15: perform the following operations: ○ selection {as selection pressure}. ○ recombination {as the pc used for selection of the features}. ○ mutation {as the pm used for selection of the features}. – step 16: generate new selected fused features with optimal weight ω. paradigmplus (2023) 4:1 24 olufemi s. ojo et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . – step 17: goto step 3 until maximum iteration is reached. – step 18: output selected fused features with optimal weight ω based on best fitness value. steps 2-14 are the procedural steps involved in standard cnn. steps 15-18 are introduced to modify and optimize the weight of cnn for performance improvement. 3.5 evaluation measures the performance of the cnn tuned with ga (cnn-ga) was done using recognition accuracy, false rejection rate (frr), computation time, and false acceptance rate (far). 4 results table 2 shows the combined outcome of cnn and cnn-ga for all matrices at a resolution of 128 by 128 pixels, with a threshold value of 0.76. regardless of the threshold value, the results show that the cnn-ga model has the shortest recognition time when compared to the cnn model. comparing the sensitivity, false positive rate, recognition accuracy, and specificity of the cnn and cnn-ga models at 128 by 128-dimensional sizes, the study found that the cnn-ga model performed better than the cnn model in terms of accuracy, specificity, and false-positive rate as shown in figures 1 and 2. table 2: cz<omparison of cnn and cnn-ga at 128 by 128-pixel resolution at 0.76 threshold value algorithm far (%) frr (%) accuracy (%) time(sec) cnn-ga 3.51 3.51 96.49 76.25 cnn 7.02 7.60 92.54 97.01 figure 1: graph of false rejection rate the improved face recognition accuracies with cnn-ga got 96.49% while the face recognition accuracies with cnn got 92.54%. it can be inferred from the results based on the performance metrics that cnn-ga applied with face gave the better result. recognition accuracies coupled with the false paradigmplus (2023) 4:1 development of an improved convolutional neural network 25 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 2: graph of accuracy acceptance rate generated with the face using cnn-ga at 0.76 threshold values were as follows: face had 96.49% accuracy at 3.51% false acceptance rate (far). while with cnn at 0.76 threshold values are as follows: face had 92.54% accuracy at 7.02% far, respectively. finally, the aforementioned results were determined based on the optimum threshold value which happened to be selected because of its outstanding performance compared to other threshold values. in view of the above results, the face with cnn-ga gave more accuracy due to a high number of true positives as well as a low number of true negatives leading to high accuracy. 5 conclusion and recommendation due to its great accuracy, a face-based attendance system would result in a more trustworthy, accurate, and secure attendance system on any repository system. in other words, the developed cnn-ga technique has ensured good classification performance and robustness against various attacks with optimal computational efficiency in terms of its accuracy and time. this study has contributed to knowledge by developing an improved attendance system that was able to produce a robust monitoring system. the results obtained show that faces with cnn-ga gave more accuracy due to a high number of true positives as well as a low number of true negatives leading to high accuracy. authors’ information – olufemi s. ojo is currently a lecturer at the department of computer sciences, ajayi crowther university oyo town in nigeria, he has a ph.d. in computer science from ladoke akintola university of technology, ogbomoso. nigeria. his research focuses on pattern recognition, optimization techniques, and e-learning. he is a full member of the international association of engineers (iaeng) and computer professionals of nigeria (cpn). – mayowa o. oyediran is a senior lecturer in the department of computer engineering at ajayi crowther university, oyo town in nigeria, he has a ph.d. in computer engineering from ladoke akintola university of technology, ogbomoso. nigeria. he is specialized in distributed systems and applications with a focus on cloud computing, fog computing, manets, etc. paradigmplus (2023) 4:1 26 olufemi s. ojo et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . – babatunde j. bamgbade is a lecturer and chief program analyst at both the federal college of forestry/forestry research institute of nigeria and the national open university of nigeria (noun). he has the following academic qualifications: b. sc, m.sc, mcs (professional), and pgd all in computer science from the university of ibadan, jabu, and ambrose alli university, respectively. he has several publications from referred and reputable journals. his areas of specialization in research are: machine learning and, software development. – emmanuel a. adeniyi is currently a lecturer and researcher in the department of computer sciences at precious cornerstone university, ibadan, nigeria. his area of research interest is information security, the computational complexity of algorithms, the internet of things, and machine learning. he has published quite a number of research articles in reputable journal outlets. – godwin nse ebong is an accomplished professional passionate about technology and innovation. he holds a masters’s degree in data science from the university of salford and a bachelor of science in engineering from caritas university. he is an expert in cloud engineering, data engineering, and data analysis, with a proven track record of developing data-driven solutions for leading organizations. godwin nse ebong is a lifelong learner, constantly seeking new challenges and opportunities to expand his knowledge and expertise. – sunday adeola ajagbe is a ph.d. candidate in the computer engineering department at the ladoke akintola university of technology (lautech), ogbomoso, nigeria. he is also a lecturer at first technical university, ibadan, nigeria. he obtained msc and bsc in information technology and communication technology respectively at the national open university of nigeria (noun), and he also has pgd in electronics and electrical engineering at lautech. he has earlier obtained hnd and nd in electrical and electronics engineering at the polytechnic, ibadan, ibadan, nigeria. his specialization includes artificial intelligence (ai), natural language processing (nlp), information security, data science, the internet of things (iot), biomedical engineering, and smart solutions. authors’ contributions – olufemi s. ojo participated in the conceptualization, validation, and supervision. – mayowa o. oyediran participated in the conceptualization, and methodology. – babatunde j. bamgbade participated in the resources, software, project administration, and writing of the original draft. – emmanuel a. adeniyi participated in the methodology, resources, and writing of the original draft. – godwin nse ebong participated in the resources, editing, and review. – sunday adeola ajagbe participated in the editing, review, visualization, and supervision. competing interests the authors declare that they have no competing interests. funding no funding was received for this project. paradigmplus (2023) 4:1 development of an improved convolutional neural network 27 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . references [1] w. r. louis, b. bastian, b. mckimmie, and a. j. lee, “teaching psychology in australia: does class attendance matter for performance?,” australian journal of psychology, vol. 68, no. 1, pp. 47– 51, 2016. [2] i. adeyanju, o. ojo, and e. omidiora, “recognition of typewritten characters using hidden markov models,” british journal of mathematics & computer science, vol. 12, no. 4, p. 1, 2016. [3] z. pei, h. xu, y. zhang, m. guo, and y.-h. yang, “face recognition via deep learning using data augmentation based on orthogonal experiments,” electronics, vol. 8, no. 10, p. 1088, 2019. [4] l. li, x. mu, s. li, and h. peng, “a review of face recognition technology,” ieee access, vol. 8, pp. 139110–139120, 2020. [5] k. m. abiodun, e. a. adeniyi, d. r. aremu, j. b. awotunde, and e. ogbuji, “predicting students performance in examination using supervised data mining techniques,” in informatics and intelligent applications: first international conference, iciia 2021, ota, nigeria, november 25–27, 2021, revised selected papers, pp. 63–77, springer, 2022. [6] s. d. liang, “optimization for deep convolutional neural networks: how slim can it go?,” ieee transactions on emerging topics in computational intelligence, vol. 4, no. 2, pp. 171–179, 2018. [7] m. pandiselvi, m. renuka, s. p. hussaima, b. shenbagam, and p. dhivya, “rfid based smart class attendance system with absentees using face verification,” asian journal of applied science and technology (ajast) volume, vol. 1, 2017. [8] j. k. adeniyi, a. e. adeniyi, y. j. oguns, g. o. egbedokun, k. d. ajagbe, p. c. obuzor, and s. a. ajagbe, “comparison of the performance of machine learning techniques in the prediction of employee,” paradigmplus, vol. 3, no. 3, pp. 1–15, 2022. [9] o. adeniji and a. o. adeniji, “a model for intrusion detection in cybersecurity using random forest algorithm,” african journal of computer and ict, vol. 14, no. 2, pp. 46–51, 2021. [10] s. d. ikhar, s. m. bhakre, and v. m. bodhe, “a review paper on: student attendance system by face detection,” international journal of advanced research in computer and communication engineering (jarcce), vol. 6, no. 1, 2017. [11] s. p. amit, s. p. shubham, a. r. kaustubh, and b. roshan, “automated attendance system using facial recognition,” 2020. [12] r. o. ogundokun, m. o. adebiyi, o. c. abikoye, t. o. oladele, a. f. lukman, a. e. adeniyi, a. a. adegun, b. gbadamosi, and n. o. akande, “evaluation of the scholastic performance of students in 12 programs from a private university in the south-west geopolitical zone in nigeria,” f1000research, vol. 8, p. 154, 2019. [13] i. d. oladipo, m. abdulraheem, j. b. awotunde, a. k. bhoi, e. a. adeniyi, and m. k. abiodun, “machine learning and deep learning algorithms for smart cities: a start-of-the-art review,” iot and ioe driven smart cities, pp. 143–162, 2021. [14] a. j. kehinde, a. e. adeniyi, r. o. ogundokun, h. gupta, and s. misra, “prediction of students’ performance with artificial neural network using demographic traits,” in recent innovations in computing: proceedings of icric 2021, volume 2, pp. 613–624, springer, 2022. [15] t. a. kumar, r. rajmohan, m. pavithra, s. a. ajagbe, r. hodhod, and t. gaber, “automatic face mask detection system in public transportation in smart cities using iot and deep learning,” electronics, vol. 11, no. 6, p. 904, 2022. paradigmplus (2023) 4:1 28 olufemi s. ojo et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [16] s. a. ajagbe, k. a. amuda, m. a. oladipupo, f. a. oluwaseyi, and k. i. okesola, “multiclassification of alzheimer disease on magnetic resonance images (mri) using deep convolutional neural network (dcnn) approaches,” international journal of advanced computer research, vol. 11, no. 53, p. 51, 2021. [17] o. d. adeniji, s. o. adeyemi, and s. a. ajagbe, “an improved bagging ensemble in predicting mental disorder using hybridized random forest-artificial neural network model,” informatica, vol. 46, no. 4, 2022. [18] s. s. bahety, v. tejaswi, k. kumar, s. r. balagar, and b. anil, “implementation of automated attendance system using facial identification from deep learning convolutional neural networks,” international journal of engineering research and technology (ijert), vol. 8, no. 15, pp. 170–174, 2020. [19] m. ahmed, m. d. salman, r. adel, z. alsharida, and m. hammood, “an intelligent attendance system based on convolutional neural networks for real-time student face identifications,” journal of engineering science and technology, vol. 17, no. 5, pp. 3326–3341, 2022. [20] s. b. joseph, e. g. dada, s. misra, and s. ajoka, “parallel faces recognition attendance system with anti-spoofing using convolutional neural network,” in illumination of artificial intelligence in cybersecurity and forensics, pp. 123–137, springer, 2022. [21] f. p. filippidou and g. a. papakostas, “single sample face recognition using convolutional neural networks for automated attendance systems,” in 2020 fourth international conference on intelligent computing in data sciences (icds), pp. 1–6, ieee, 2020. [22] s. sawhney, k. kacker, s. jain, s. n. singh, and r. garg, “real-time smart attendance system using face recognition techniques,” in 2019 9th international conference on cloud computing, data science & engineering (confluence), pp. 522–525, ieee, 2019. [23] t.-v. dang, “smart attendance system based on improved facial recognition,” journal of robotics and control (jrc), vol. 4, no. 1, pp. 46–53, 2023. paradigmplus (2023) 4:1 introduction related work methodology data acquisition data pre-processing conversion to grayscale image normalization thinning data augmentation feature extraction and classification phase using deep learning approach training and classification evaluation measures results conclusion and recommendation vol. 2, no. 3, pp 37 52, 2021 doi: 10.55969/paradigmplus.v2n3a3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ultra-short heart rate variability and poincaré plots jose gallardo id 1, 5�, giannina bellone id 2, 3, 5, and marcelo risk id 1, 4, 5 1universidad tecnologica nacional, buenos aires, argentina jgallardo@frba.utn.edu.ar 2department of science and technology, laboratory of chronobiology, universidad nacional de quilmes, buenos aires, argentina 3chronophysiology lab., institute for biomedical research (biomed), universidad católica argentina, buenos aires, argentina gbellone@uca.edu.ar 4instituto de medicina traslacional e ingeniería biomédica (imtib), buenos aires, argentina marcelo.risk@hospitalitaliano.org.ar 5consejo nacional de investigaciones científicas y técnicas (conicet), buenos aires, argentina abstract the short-term heart rate variability records of 5 minutes are currently considered very extensive compared to other indicators of health and well-being of people, such as body weight, blood glucose, blood pressure, etc. to find records of indicators less than 5 minutes, ultra-short heart rate variability, equivalent to the short-term heart rate variability, samples registers of 5 minutes is used to extract ss (stress score index) and s / ps (sympathetic/parasympathetic ratio) from the poincaré diagram and correlate with the time and frequency domain of a group of subjects. to do this, the electrocardiographic signals were recorded for 300 seconds at rest then, the cardiotachograms (rr series) were calculated and used as a gold standard, and cardiotachogram were also calculated at intervals of 60, 90, 120, 180, and 240 seconds to perform the analysis of concordance with the gold standard. we use concordance analysis: spearman, bland, and altman correlation, and cliff’s delta. this study set out to, that the indicator s/ps – proposed for the analysis of the autonomic balance – could be regarded as an alternative for traditionally be used lf/hf (low frequency / high frequency) of the frequency domain or sd21 (sd2 / sd1) of the graphic poincaré domain. in addition, the ss times within records of 120 seconds (rho = 0.90, p = 3.28e − 06), were equivalent to those of short-term hrv and s/ps of 90 seconds (rho = 0.93, p = 3.15e−06). this finding, while preliminary, suggests that a reduction in measurement times will make it possible to increase the use of heart rate variability analysis. in future investigations, it might be possible to use these indicators in different populations and use of different pre-processing methods. keywords: heart rate variability ⋅ ultra-short heart rate variability ⋅ poincaré ⋅ ss ⋅ s/sp received: 1 november 2021 ⋅ accepted: 29 december 2021 ⋅ published: 31 december 2021. 1 introduction heart rate variability (hrv) reflects cardiac modulation by sympathetic and vagal components of the autonomic nervous system (ans) that produces beat-to-beat changes. these changes are analyzed by recording the electrocardiogram (ecg) in whose tracing r waves are identified. the time distance this article is distributed under the terms of the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. https://doi.org/10.55969/paradigmplus.v2n3a3 https://orcid.org/0000-0002-3412-7478 https://orcid.org/0000-0003-1590-5307 https://orcid.org/0000-0003-0107-2551 38 jose gallardo et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 1: determination of rr timings from ecg signal. between two consecutive r waves allows obtaining a series of time intervals called cardiotachogram [1] (see figure 1). hrv is used as an indicator parameter of the level of cardiovascular health, since it is possible to study the changes in the sympathetic-vagal balance of the cardiac response, resulting in a very useful tool in the study and analysis of the evaluation of individuals who perform sports activities [2, 3]. hrv is a non-invasive method that is used as an indicator of cardiovascular health. in studies of diseases such as diabetes and analysis of cardiovascular and cerebrovascular events, in hypertensive patients, detection of drowsiness, in sleep deprived subjects [4], stressful situations and recently in stress during sports training and in subsequent physiological recovery [5]. hrv studies are classified into long-term studies, with 24-hour records [6] and short-term studies, with 5-minute records [1, 7, 8, 9]. the increase in the use of mobiles devices allowed the appearance of applications for the analysis of the health status of people, which perform measurements of heart rate, blood sugar, body mass, etc. in short times. due to this, hrv measurement times should be less than 5 minutes [10] to increase its use as a diagnostic tool in the medical clinic and and also achieve almost real-time measurements [11]. the methods currently used in short-term hrv are geometric and nonlinear analysis [1], and to a greater extent, the time domain analysis using a statistical approach, by means mean of rr intervals (rrm), standard deviation of rr intervals (sdnn), root mean square of the differences between consecutive rr intervals (rmssd) and percentage value of frequency of successive differences of rr intervals that spanned more than 50 ms (pnn50) (see figure 2). the frequency domain analysis by power spectral density (psd) using lomb periodogram (lp) or fast fourier transform (fft) for non-equispaced rr series, lp does not require re-sampling of the series, unlike fft method [12, 13] (see figure 3). frequency domain indexes can use the following frequency bands: low frequency (lf) (0.04-0.15 hz), high frequency (hf) (0.15-0.4 hz), normalized parameters (lfn.u.), (hfn.u.) and lf / hf ratio [14] (see figure 3). the poincaré domain geometric indexes describe the correlation between consecutive rr intervals [15]. cross axis, the (sd1) index reflects the short-term variability of heartbeats, related to parasymparadigmplus (2021) 2:3 ultra-short heart rate variability and poincaré plots 39 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 2: cardiotachogram on healthy subjects showing time-domain indexes. the abscissa and ordered axes represent time in seconds and rr intervals in ms, respectively. figure 3: spectral analysis on healthy subjects showing indexes’ values in the frequency domain. the abscissa and ordered axis present the psd and the frequency, respectively. pathetic activity. the (sd2) index reflects the long-term variability, and the (sd2/sd1) ratio is a measure of the relation between the long-and short-term variability similar to the lf/hf ratio by its analogy and properties [4]. from the poincaré domain, for the assessment of autonomic balance in soccer players [16] proposed two new indexes, of short hrv, a direct index for the evaluation of sympathetic activity ss (stress score) and s/ps (sps) (sympathetic-parasympathetic ratio) as an indicator of autonomous balance. the reduction of the recording times of the ss and sps indicators can increase the use of hrv and particularly its application in the evaluation of sports performance in athletes and also improve the knowledge of the physiological meaning of hrv. even though, studies suggest minimum lengths of the rr series depending on the index to be calculated [10]. therefore, this study set out on the one hand, to analyze the behavior of ss and sps with respect to other time and frequency domain indexes, and on the other hand, to find the minimum recording times for ultra-short hrv required so that the indexes ss and sps are equivalent to those obtained using short hrv. paradigmplus (2021) 2:3 40 jose gallardo et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 material and methods 2.1 subjects and design overall, twenty-three subjects with a mean age of 23 years voluntarily participated in this study. the subjects were healthy, nonsmokers and had no known cardiovascular diseases and free from any medication. every subject provided written informed consents for participation in this study before starting it. this study went through ethics approval from the research committee at national university of quilmes, buenos aires, argentina (04/07/2016/no. 3). 2.2 hrv analysis the ecg signal of each subject was obtained in resting position, for 5 min (300 s) and 2 min of stabilization time, with a holter, by means of the di lead (di lead, is formed by placing electrodes between the right arm and the left arm), with a sampling frequency of 225 hz. ecg signal was processed, using a notch filter was applied to remove interference from the power supply network (50 or 60 hz), the trends were eliminated by polynomials of order 6 previously adjusted to the signal and then oversampled at 1 khz, to increase the reliability of measures of the r wave. the modified pan-tompkins algorithm was used [17] to detect r peaks, in the qrs complex of the ecg, then the rr time interval series was calculated. in this series, the linear trend and the zero-frequency component were eliminated. to detect atypical beats such as the non-sinus, ectopics, false signal distortions, muscles movement, electromagnetic interference, etc. [1] an impulse rejection filter was used [18, 19]. these atypical beats were replaced by interpolation by another interval whose value was the average of the six rr intervals neighboring the one considered artifact. [20, 21]. for the purposes of the ultra-short hrv analysis, to each 300 s series (the "gold standard" (gs) series) time windows of 60, 90, 120, 180 and 240 seconds were extracted (see figure 4). the hrv analysis was performed in the time domain and the lp was used for the psd in the frequency domain. figure 4: rr series graphic representation with different time lengths in seconds starting from the beginning. the series of 300s corresponds to the gold standard. 2.3 poincaré plot the poincaré graph presented in figure 5 is a geometric method where by plotting the rrn interval versus the next rrn+1 interval and adjusting the graph to an ellipse allows us to calculate the hrv paradigmplus (2021) 2:3 ultra-short heart rate variability and poincaré plots 41 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 5: poincaré graph. the abscissa represents the time interval rrn, the ordinate the next interval rrn+1, respectively. the fit to an ellipse, line y = x, sd1 parasympathetic activity and sd2 sympathetic activity indicators. indexes [22]. the poincaré plot method allows to obtain the standard deviation of the instantaneous variability of rr interval, minor axis of the ellipse sd1 (parasympathetic activity) and the standard deviation of the continuous variability of the long-term rr interval, major axis of the ellipse sd2 (sympathetic activity) and the ratio sd2/sd1 a measure of the relation between the long-term and short-term variability (sd21) [23]. equation 1, equation 2: sd1 = √ v ar( rrn − rrn+1 √ 2 ) (1) sd2 = √ v ar( rrn + rrn+1 √ 2 ) (2) where rrn is a certain interval, rrn+1 is the next interval and var represents the variance. the parameters ss and sp s were calculated from the poincaré diagram using the equations proposed by [16], equation 3 and equation 4 . ss = 1000 sd2 (3) sp s = ss sd1 (4) signal processing and hrv calculations were developed in gnu octave. paradigmplus (2021) 2:3 42 jose gallardo et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 statistical analysis the algorithm proposed by [24], for the situation of lack of normality of some of the series and the concurrent validity criterion applied by [10] were used for the equivalence analysis between the gs series and each of the ultra-short series belonging to the same sample. spearman correlation coefficients (rho) [23] were calculated for the ss and sps indicators between the gs series and each of the 60, 90, 120, 180 and 240 s series, with which the number of components of each series (rr intervals) does not prevent the correlation calculation, since for each series there are 23 values (23 study subjects) for each index. using the bland-altman method of graphical analysis [25], the degree of agreement between the gs and each of the ultrashort series for each index was established. in addition, the cliff delta, (d) [26], was used to quantify the effect size of the bias in ss and sps, hrv parameters, measurements between ultra-short series and gs serie. the value of d can be between 1 and -1 and is considered for ∣d∣< 0.15 an "insignificant" effect size, ∣d∣< 0.33 "small", ∣d∣< 0.47 "medium" and "large", for another values [26]. the relative error percentage between the gs and each one of the series was also calculated, using equation 5 for ss and equation 6 for the sps index, respectively [27]. ess(%)= median(gsss)− median(seriexss) median(gsss) ∗ 100 (5) esp s(%)= median(gssp s)− median(seriexsp s) median(gssp s) ∗ 100 (6) where x = 60, 90, 120, 180 and 240. values of p < 0.05, were considered statistically significant. the r software was used to execute the statistical calculations. 3 results the result of the processing of the ecg signals extracted from the participants, the calculation of the rr series and their pre-processing can be analyzed from figure 6, where figure 6(a) shows the ecg (a) obtained at rest position (b) after applying of the notch filter, elimination of the nonlinear trend of the ecg and oversampled to 1khz figure 6: ecg signal of one subject. paradigmplus (2021) 2:3 ultra-short heart rate variability and poincaré plots 43 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (a) r peaks detection from ecg signal (b) series rr calculated from ecg with outliers figure 7: processing of the ecg signal. signal, corresponding to one subject participating in the study, extracted from the holter memory sampled at 225 hz, visibly distorted by noise. subsequently, the signal processing is carried out in figure 6(b), where the result of the application of the notch filter is observed, the elimination of the nonlinear trend in the signal and its subsequent oversampled at 1 khz. the results in the processing of the ecg signal to identify the peaks of the r waves and to this filtered signal are shown in figure 7(a), while figure 7(b) shows the rr series obtained, where some atypical beats are observed. these artifacts were later corrected by interpolation. figure 8(a) and figure 8(b) respectively show the pre-processed series and without its linear trend and eliminating the continuous component. table 1 shows the result of the shapiro-wilk normality test for each of the rr time series of the ss and sps indicators. table 2 presents the spearman correlations of the gs series, on the one hand between the ss index and time (sdnn, rmssd) and the frequency domain (lf, hf, lf/hf) and on (a) interpolation of anomalous beats (b) linear trend and the zero-frequency component were eliminated of the series figure 8: interpolation of the ecg signal. paradigmplus (2021) 2:3 44 jose gallardo et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 1: shapiro-wilk normality test p value. rr series ss sps 60 0.01 0.00 90 0.04 0.00 120 0.09 0.01 180 0.02 0.01 240 0.01 0.01 300 0.14 0.00 the other the correlation between these indexes and the sps index. the observed correlation between lf/hf and sps suggests a similar behavior for both indexes, as does the correlation between sd21 and sps. table 2: spearman correlation between ss, sps indexes and time sdnn, rmssd and frequency domain lf, hf, lf/fh indexes corresponding to gs series. *p < 0.05, **p < 0.005 indexes ss sps rmssd -0.78** -0.86** sdnn -0.77** -0.77** lf -0.50* -0.46* hf -0.72** -0.78** lf/hf 0.39 0.60** sd21 0.41 0.62** table 3 shows the ss and sps indicators of the rr series, whose values are given as medians and quartiles (1st; 3rd). table 3: ss and sps indicators of rr series, median (1st; 3rd) quartiles. rr series ss sps 60 13.61 (10.63 21.12) 0.39 (0.21 1.26) 90 14.10 (11.48 17.36) 0.48 (0.23 1.07) 120 13.48 (10.53 17.16) 0.47 (0.24 1.05) 180 13.18 (9.92 17.62) 0.47 (0.23 1.09) 240 13.47 (10.07 17.88) 0.45 (0.24 1.01) 300 12.87 (10.10 16.85) 0.43 (0.25 0.96) the correlation analysis of the series gs vs 120 s of the ss index is observed in the figure 9. the analysis of the concordance between the series of 120 s and the gs by means of the bland-altman plot is shown in figure 10, where the lines of bias are observed, the median of the difference between the series and the 2.5 and 97.5 percentiles of the loa 95 %. gs is represented on the abscissa axis, not as traditionally, the average of the measurements [28]. the concordance analysis between gs and different time series of the ss index is presented in table 4. where an increase in the values of the spearman correlation coefficients (rho) is observed with the temporal increase of the ultra-short series with value of 0.90 from the series of 120 s of ss index. the resulting cliff delta (d) values are very small and considered insignificant for all series. values of % error (ess) that are not greater than 10 % are observed between the gs serie and each of the ultraparadigmplus (2021) 2:3 ultra-short heart rate variability and poincaré plots 45 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0 10 20 30 5 10 15 20 25 gs s s _ 1 2 0 s figure 9: correlation analysis of gs vs ss 120 s of the ss index, rho = 0.90, p = 3.28e-06. biasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbias loaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloa loaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloa −6 −4 −2 0 2 0 10 20 30 gs g s − s s _ 1 2 0 s bland−altman figure 10: analysis of the bland-altman plot to study the concordance between the gs and the 120 s series of the ss index. short series. an overestimation of gs is observed due to the negative result of the bias. and there is a decrease in the loa 95% with the temporary increase of the ultra-short series. paradigmplus (2021) 2:3 46 jose gallardo et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 4: analysis of concordance between gs and different time series of the ss index. ss rr series (s) correlation (rho) (p) cliff (d) (c.i. 95%) % error (ess) wilcoxon (p) bias (1st; 3rd) loa 95% 300 vs 60 0.71 (1.98e-04) 0.14 (-0.20 0.45) 5.74 0.04 -1.51 (-3.90 0.57) -14.65 5.10 300 vs 90 0.84 (1.71e-06) 0.16 (-0.18 0.48) 9.48 0.02 -1.04 (-2.56 0.35) -6.67 1.89 300 vs 120 0.90 (3.28e-06) 0.09 (-0.25 0.41) 4.66 0.09 -0.56 (-1.54 0.46) -5.78 2.27 300 vs 180 0.96 (2.45e-06) 0.06 (-0.28 0.38) 2.37 0.03 -0.58 (-1.36 0.35) -3.90 1.61 300 vs 240 0.98 (1.99e-06) 0.06 (-0.28 0.38) 4.61 0.01 -0.48 (-0.95 -0.07) -3.23 1.40 for the sps index, figure 11 shows the result of the correlation between the gs series and the ultrashort series of 90 s an accumulation of values below 0.5 of the gs series is observed in the graph. the analysis of the concordance of the gs and sps of 90 s carried out using the bland-altman graph is shown in figure 12 where it is possible to observe an overestimation of the gs due to the negative value resulting from the bias in addition to an accumulation of negative values between 0 and 0.5 and a large asymmetry between the loa. table 5 shows the results of the methods used for the analysis of agreement between the gs and the ultra-short series corresponding to the sps indicator. high spearman values are observed mainly 0.0 0.5 1.0 1.5 2.0 2.5 0.0 0.5 1.0 1.5 2.0 2.5 gs s p s _ 9 0 s figure 11: correlation analysis of gs vs sps 90 s of sps index, rho = 0.93, p = 3.15e-06. paradigmplus (2021) 2:3 ultra-short heart rate variability and poincaré plots 47 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . biasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbiasbias loaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloa loaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloaloa −0.4 0.0 0.4 0.0 0.5 1.0 1.5 2.0 2.5 gs g s − s p s _ 9 0 s bland−altman figure 12: analysis of the bland-altman plot to study the concordance between the gs and the 90 sseries of the sps index. table 5: analysis of concordance between gs and different time series of the sps index. sps rr series (s) correlation (rho) (p) cliff (d) (c.i. 95%) % error (esp s) wilcoxon (p) bias (1st; 3rd) loa 95% 300 vs 60 0.80 (7.02e-06) 0.01 (-0.33 0.34) 9.63 0.52 -0.02 (-0.15 0.08) -1.04 0.74 300 vs 90 0.93 (3.15e-06) 0.05 (-0.28 0.38) 10.15 0.15 -0.02 (-0.14 0.06) -0.51 0.13 300 vs 120 0.95 (2.79e-06) 0.03 (-0.30 0.35) 9.48 0.71 0.00 (-0.09 0.06) -0.22 0.18 300 vs 180 0.98 (2.05e-06) 0.05 (-0.29 0.372) 8.85 0.29 -0.02 (-0.08 0.03) -0.32 0.22 300 vs 240 0.99 (1.71e-06) 0.06 (-0.28 0.38) 4.91 0.20 -0.03 (-0.06 0.01) -0.26 0.23 from the correlation between gs and the series of 90 s, rho = 0.93, p = 3.15e-06, reaching a value of rho = 0.99, p = 1.71e-06, for the correlation gs vs 240 s. the cliff delta (d) shows negligible differences between the gs series, patron, and each of the ultra-short series of 60, 90, 120, 180 and 240 s. in the case of the % error (esp s), the is less than 10% in all cases. an overestimation of the gs series is observed, given by the negative value of the bias, in addition the wilcoxon test shows that the value of bias could eventually be null. from the analysis of the bland-altman graph, it appears that the loa 95% values decrease with the temporal increase of the ultra-short series, showing a greater degree of concordance between these series and the gs. paradigmplus (2021) 2:3 48 jose gallardo et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 discussion the poincaré graph is a simple method for evaluating hrv in the short term (5min) where, in addition, atypical rr intervals (premature beats, compensatory pauses or technical artifacts such as disconnection of electrodes, electromagnetic), influence the poincaré trace with less force than in the analysis of the temporal and frequency domains [23, 29, 30]. the parameters extracted from the diagram sd1, sd2, separately, they are related to the linear domains of time and frequency [23, 31]. however, the relationship between the two, sd21, could be considered as an alternative measure to the autonomic balance indicator in the frequency domain lf/hf [31, 32, 4], and also presents a high correlation with non-linear hrv parameters, both in sick and healthy subjects [33]. although it is not possible to establish a due metric mainly due to large inter-individual differences and statistical effects are shown on the group level [34, 29, 4]. the indicators ss and sps, proposed by [16] provide reference values for the analysis of hrv in the short-term in elite soccer players and propose a interpretation to understand the physiological meaning of autonomic balance. previous studies that have pointed out the importance of the reduction in measurement times to increase the use of hrv, especially in the daily clinic and analysis of sports performance in athletes. using ultra-short hrv, records less than 5 min, [35] analyzed the sd21 indicator, establishing a minimum time required for its registration and [36] also using ultrashort hrv, they determined the minimum times for ultra-short registrations for ss and sps. the correlations found in this work, suggest on the one hand that sps could be considered as an alternative for sd21 and also for lf/hf [37] and on the other that the behaviour of ss is similar to that found by [16]. for the analysis of the indicators, a method was used to reliably evaluate the equivalences between the series of the ultra-short hrv ss and sps indicators and the short-term, 5 min. a value of rho = 0.90 was adopted as the cutoff criterion, as a minimum, this threshold ensures that a measurement represents at least 81 % of the variability (rho2 = 0.81) an extremely low risk of committing a type 1 error [10]. the results show a similarity in the values of the medians in the series of each index except for the series of 60 s, this is possibly due to the fact that this index requires a longer time window [29], in addition a positive increase in the values that reflect levels of agreement between the ss series and also in the series of the sps indicator. the threshold value proposed for the correlation in the ss index is observed from the 120 s series, negligible values are also observed for the delta of cliff and very low values of the error (ess). the ultra-short series of 180 and 240 s show a similar behaviour. the analysis of the bland-altman graphs show high concordance from the gs series and the 120 s series, evidencing the similarity between the gs series and the ultra-short 180 s and 240 s, respectively. this allows to establish the equivalence of the series of 120 s with the gs for the index ss. the analysis of the sps indicator shows that from the 90 s series the criterion used is fulfilled, where rho = 0.93, greater rho = 0.90, here also the values of cliff’s delta are negligible, showing sps equivalence with the gs from the ultra-short series of 90 s. these findings can help us to understand, on the one hand, the usefulness of the poincaré plot in the analysis of hrv, on the other hand, the decrease in recording time may allow an increase in the use of hrv for the analysis of the well-being of people. these data should be interpreted with caution because the study had a small sample size. on the other hand, different stabilization times prior to the records were not analyzed. furthermore, only the supine position was used for the ecg recordings and only the segment of the onset of the signal was analyzed. on the other hand, a single rr series pre-processing method was used. it will be necessary to carry out more studies that take these variables into account. paradigmplus (2021) 2:3 ultra-short heart rate variability and poincaré plots 49 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 conclusions in conclusion, the new sps indicator proposed for the analysis of autonomic equilibrium could be considered as an alternative to the traditional lf / hf. furthermore, the ultra-short recordings of 120 s for ss, and 90 s for sps, were equivalent to shortterm hrv (5 min). the reduction in measurement times could possibly help increase the use of hrv to monitor the health and well-being and especially in the sports evaluation of large groups of athletes, by reducing the recording and processing times of the signals. in future investigations, it might be possible to use these indicators in different populations and the use of different pre-processing methods. authors’ information – josé gallardo studied electronics engineering at the universidad tecnologica nacional facultad buenos aires, where he presented his doctorate thesis in engineering. he also obtained a master’s degree in biomedical engineering from the universidad nacional de entre rios. he is a principal professional at the consejo nacional de investigaciones cientificas y tecnicas (conicet) as well as professor and researcher at the universidad de la marina mercante in buenos aires, argentina. – giannina bellone has a degree in psychology from the universidad catolica argentina and a degree in journalism. she is currently finishing her phd in science and technology at the universidad nacional de quilmes, argentina. she is also a specialist in prevention of occupational and road accidents and a researcher in the chronophysiology laboratory of biomed-ucaconicet. she is professor of labor psychology in the psychology career of the uca. – marcelo risk is currently director of the instituto de medicina traslacional e ingenieria biomedica associated with the consejo nacional de investigaciones cientificas y tecnicas (conicet), istituto universitario del hospital italiano de buenos aires and the hospital italiano de buenos aires, argentina. he is also full-time professor at the biomedical engineering program at the instituto universitario hospital italiano de buenos aires. mr. marcelo risk has a phd from the universidad de buenos aires, a scd in bioengineering from the universidad nacional de córdoba, an mba from the universidad de palermo, and a degree in electronics engineering from the universidad tecnologica nacional, argentina. authors’ contributions – josé gallardo participated in the research, methodology, software development as well as writing and reviewing the manuscript. – giannina bellone participated in the methodology, taking measurements, and reviewing the manuscript. – marcelo risk participated in the research, methodology, writing, and reviewing manuscript. competing interests the authors declare that they have no competing interests. funding no funding was received for this project. paradigmplus (2021) 2:3 50 jose gallardo et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . references [1] m. malik, heart rate variability. new york: futura publishing company inc., 1 ed., 1995. [2] c. r. bellenger, j. t. fuller, r. l. thomson, k. davison, e. y. robertson, and j. d. buckley, “monitoring athletic training status through autonomic heart rate regulation: a systematic review and meta-analysis,” sport. med., vol. 46, no. 10, pp. 1461–1486, 2016. [3] s. jiménez morgan and j. a. molina mora, “effect of heart rate variability biofeedback on sport performance, a systematic review,” appl. psychophysiol. biofeedback, vol. 42, no. 3, pp. 235–245, 2017. [4] j. gallardo, g. bellone, s. plano, d. vigo, and m. risk, “heart rate variability: influence of preprocessing methods in identifying single-night sleep-deprived subjects,” j. med. biol. eng., pp. 1–7, 2021. [5] m. a. maggioni, l. c. rundfeldt, h. c. gunga, m. joerres, g. merati, and m. steinach, “the advantage of supine and standing heart rate variability analysis to assess training status and performance in a walking ultramarathon,” front. physiol., vol. 11, no. july, 2020. [6] m. risk, j. sobh, r. barbieri, r. armentano, a. ramirez, and j. saul, “variabilidad de las señales cardiorespiratorias. parte 2: variabilidad a largo plazo,” rev. argentina bioingenieria, vol. 2, no. 2, 1996. [7] m. risk, j. sobh, r. barbieri, r. armentano, a. ramirez, and j. saul, “variabilidad de las señales cardiorespiratorias. parte 1: variabilidad a corto plazo,” rev. argentina bioingenieria, vol. 2, no. 1, 1996. [8] l. g. gamero, m. risk, j. f. sobh, a. j. ramirez, and j. p. saul, “heart rate variability analysis using wavelet transform,” comput. cardiol., vol. 0, no. 0, pp. 177–180, 1996. [9] m. risk, v. bril, c. broadbridge, and a. cohen, “heart rate variability measurement in diabetic neuropathy: review of methods,” diabetes technol. ther., vol. 3, no. 1, pp. 63–76, 2002. [10] f. shaffer, s. shearman, and z. m. meehan, “the promise of ultra-short-term (ust) heart rate variability measurements,” biofeedback, vol. 44, no. 4, pp. 229–233, 2016. [11] s. massaro and l. pecchia, heart rate variability (hrv) analysis: a methodology for organizational neuroscience, vol. 22. sagepub, 2019. [12] p. laguna, g. b. moody, and r. g. mark, “power spectral density of unevenly sampled data by least-square analysis: performance and application to heart rate signals,” ieee trans. biomed. eng., vol. 45, no. 6, pp. 698–715, 1998. [13] b. s. saini, d. singh, m. uddin, and v. kumar, “improved power spectrum estimation for rrinterval time series,” int. j. electr. comput. energ. electron. commun. eng., vol. 2, no. 10, pp. 154– 158, 2008. [14] j. camm, “guidelines heart rate variability,” eur. heart j., vol. 17, pp. 354–381, 1996. [15] u. bracale, m. rovani, m. bracale, g. pignata, f. corcione, and l. pecchia, “totally laparoscopic gastrectomy for gastric cancer: meta-analysis of short-term outcomes,” minim. invasive ther. allied technol., vol. 21, no. 3, pp. 150–160, 2012. paradigmplus (2021) 2:3 ultra-short heart rate variability and poincaré plots 51 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [16] n. orellana, j. n. orellana, b. de la cruz torres, e. s. cachadiña, m. de hoyo, and s. d. cobo, “" two new indexes for the assessment of autonomic balance in elite soccer players " article title: two new indexes for the assessment of autonomic balance in elite soccer players,” int. j. sport. physiol. perform. j. int. j. sport. physiol. perform., pp. 452–457, 2014. [17] l. sathyapriya, l. murali, and t. manigandan, “analysis and detection r-peak detection using modified pan-tompkins algorithm,” in proc. 2014 ieee int. conf. adv. commun. control comput. technol. icaccct 2014, vol. 978, pp. 483–487, 2015. [18] j. mcnames, t. thong, and m. aboy, “impulse rejection filter for artifact removal in spectral analysis of biomedical signals.,” proc. 26th annu. int. conf. ieee embs, san fr. ca, usa, vol. 1, pp. 145–8, 2004. [19] r. a. thuraisingham, “preprocessing rr interval time series for heart rate variability analysis and estimates of standard deviation of rr intervals,” comput. methods programs biomed., vol. 83, no. 1, pp. 78–82, 2006. [20] d. wejer, d. makowiec, z. et. al. struzik, and m. żarczyńska-buchowiecka, “impact of the editing of patterns with abnormal rr intervals on the assessment of heart rate variability,” acta phys. pol. b, vol. 45, no. 11, p. 2103, 2014. [21] a. e. aubert, d. ramaekers, and f. beckers, “analysis of heart rate variability in unrestrained rats. assessment of method and results.,” med. biol. eng. comp., vol. 60, pp. 197–213, 1999. [22] m. p. tulppo, t. h. makikallio, t. e. s. takala, t. seppanen, and h. v. huikuri, “analysis of heart rate dynamics during exercise,” am. j. physiol. circ. physiol., pp. 244–252, 1996. [23] p. guzik, j. piskorski, t. krauze, r. schneider, k. h. wesseling, a. wykretowicz, and h. wysocki, “correlations between the poincaré plot and conventional heart rate variability parameters assessed during paced breathing,” j. physiol. sci., vol. 57, no. 1, pp. 63–71, 2007. [24] l. pecchia, r. castaldo, l. montesinos, and p. melillo, “are ultra-short heart rate variability features good surrogates of short-term ones? state-of-the-art review and recommendations,” healthc. technol. lett., vol. 5, no. 3, pp. 94–100, 2018. [25] j.m.bland and d.g.altman, “statistical methods for assessing agreement between two methods of clinical measurement,” lancet, vol. 1, no. 8476, pp. 307–310, 1986. [26] n. cliff, ordinal methods for behavioral data analysis. psychology press, 1st ed., mar 2014. [27] m. nardelli, a. greco, j. bolea, g. valenza, e. p. scilingo, and r. bailon, “reliability of lagged poincaré plot parameters in ultra-short heart rate variability series: application on affective sounds,” ieee j. biomed. heal. informatics, vol. 2194, no. c, pp. 1–1, 2017. [28] m. l. munoz, a. van roon, h. riese, c. thio, e. oostenbroek, i. westrik, e. j. c. de geus, r. gansevoort, j. lefrandt, i. m. nolte, and h. snieder, “validity of (ultra-)short recordings for heart rate variability measurements,” plos one, vol. 10, no. 9, pp. 1–15, 2015. [29] f. shaffer and j. p. ginsberg, “an overview of heart rate variability metrics and norms,” front. public heal., vol. 5, no. september, pp. 1–17, 2017. [30] j. piskorski and p. guzik, “geometry of the poincaré plot of rr intervals and its asymmetry in healthy adults,” physiol. meas., vol. 28, no. 3, pp. 287–300, 2007. [31] c. h. hsu, m. y. tsai, g. s. huang, t. c. lin, k. p. chen, s. t. ho, l. y. shyu, and c. y. li, “poincaré plot indexes of heart rate variability detect dynamic autonomic modulation during general anesthesia induction,” acta anaesthesiol. taiwanica, vol. 50, no. 1, pp. 12–18, 2012. paradigmplus (2021) 2:3 52 jose gallardo et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [32] x. chen, r. yao, g. yin, and j. li, “consecutive ultra-short-term heart rate variability to track dynamic changes in autonomic nervous system during and after exercise.,” physiol. meas., vol. 38, no. 7, pp. 1384–1395, 2017. [33] r. a. hoshi, c. m. pastre, l. c. m. vanderlei, and m. f. godoy, “poincaré plot indexes of heart rate variability: relationships with other nonlinear variables,” auton. neurosci. basic clin., vol. 177, no. 2, pp. 271–274, 2013. [34] g. ernst, “heart-rate variability—more than heart beats?,” front. public heal., vol. 5, no. september, pp. 1–12, 2017. [35] j. m. gallardo, g. bellone, r. acevedo, and m. risk, “ultra-short-term heart rate variability analysis: comparison between poincare and frequency domain methods,” ieee lat. am. trans., vol. 20, no. 1, pp. 180–188, 2022. [36] j. gallardo, g. bellone, and m. risk, “heart rate variability: validity of autonomic balance indicators in ultra-short recordings,” in international conference on applied informatics, pp. 303–315, springer, 2021. [37] j. m. gallardo and l. n. siri, “simulador de cardiotacogramas para validación de análisis en variabilidad de la frecuencia cardíaca,” mecánica computacional, vol. 30, no. 44, pp. 3403–3412, 2011. paradigmplus (2021) 2:3 introduction material and methods subjects and design hrv analysis poincaré plot statistical analysis results discussion conclusions vol. 3, no. 2, pp 1 10, 2022 doi: 10.55969/paradigmplus.v3n2a1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . societal xr—a vision paper daniel görlich id 1,� 1srh university heidelberg, ludwig-guttmann-str. 6, 69123 heidelberg, germany daniel.goerlich@srh.de abstract most current research and developments in the field of extended reality follow a rather onesided agenda: creating virtual content, virtual worlds, and even metaverses that require all users to wear suitable head-mounted displays. they thereby reduce the application of xr to limited spaces and (relatively) few users and exclude users such as preschool children and people with certain disabilities. only a few approaches such as spatial augmented reality and social xr overcome at least some of these limitations. so far, a vision for a truly public xr that can be used by everyone without wearing technology on their bodies, is missing. this vision paper, therefore, wants to give xr research and development an additional new direction. it coins the term "societal xr" for a form of extended reality that moves technology into the environment, becomes accessible to everyone including children and people with disabilities, and integrates virtuality into their users’ reality, into their everyday life, and into public spaces. keywords: societal xr ⋅ extended reality ⋅ virtual reality ⋅ ambient intelligence ⋅ ubiquitous computing received: 20 june 2022 ⋅ accepted: 15 august 2022 ⋅ published: 17 august 2022. 1 introduction extended reality can be used in public spaces, it can be seamlessly integrated into our everyday life, and it has the potential to change entire societies. humans are social beings. as such, they desire relationships with other people and a sense of belonging to one or more groups. being part of such groups is essential for their identity finding, self-esteem, and individual well-being. in this respect, it is no wonder that most people look for social contacts wherever they are—leaving aside one or the other to escape from reality. from the very beginning of the internet, its users have always looked for opportunities to socialize in virtual spheres, whether in vzones on the early internet, in massively multiplayer online roleplaying games (mmorpg’s), in chat rooms, or in precursors of what is now called a metaverse, e.g., in lucasfilm’s habitat or linden lab’s second life [1]. however, if we analyze the current direction of virtual, augmented, mixed, and extended reality in general, we notice an extremely one-sided orientation: researchers and companies alike are striving for small or large virtual worlds, for tinyverses [2] and metaverses [3], which, however, are mostly virtual spaces. usually, three basic assumptions are made: 1○ it is always the user who enters such a virtual space instead of having it come to him or continuously surround him. this article is distributed under the terms of the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. https://doi.org/10.55969/paradigmplus.v3n2a1 https://orcid.org/0000-0002-4275-5466 2 daniel görlich . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2○ in order to participate, every user needs wearable technology such as head-mounted displays, controllers, and possibly trackers. 3○ all these virtual spaces are limited—sometimes to several million people (massively multiuser), but never so large that the entire population of a big country or even the whole world could participate. also, those people who cannot wear the technology they need on their bodies, such as some people with disabilities, young children, or people who cannot cope with the technology, such as the elderly, are always left out. approaches to overcome these three limitations are hardly explored. social xr [4] allows several people or a group of people to experience social presence and co-presence by engaging in activities and real-time conversations with each other. in some cases, the participants wear head-mounted displays, are almost completely shielded from their real environment, and are embodied by avatars [5], accepting that the use of avatars can change the user’s own and other users’ behavior [6]; in other social spaces, the participants are recorded by the camera so that photos or live videos of them are integrated into virtual environments. however, these social spaces are always located in a virtual world. spatial augmented reality; on the contrary, takes place in real environments [7]. it allows individual users and relatively small groups of participants, up to a few thousand people, to experience augmented reality in a limited but real space, usually in one common location. it does not require any participant to wear technology on their body, because the necessary equipment, at least powerful projectors and speakers, if necessary, are integrated into the environment instead. for this reason; however, it is not or only slightly interactive. figure 1 shows a classification of the virtual, augmented, and mixed reality, social xr, and spatial augmented reality paradigms on a three-dimensional diagram with the axes: – technology mainly in the environment vs technology mainly worn on the body – anchored in virtual space vs in real space – the number of potential users ranging from individual users to a small group, a massively multiuser experience, and availability for everybody. figure 1: three-dimensional categorization of xr paradigms with respect to technology (mainly on or off the user’s body), space (virtual or real), and accessibility (number of potential users). societal xr is highlighted in green. paradigmplus (2022) 3:2 societal xr a vision paper 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 1 makes it clear that although there are xr approaches for the presence of users in both virtual and real spaces, almost all of those approaches only allow users to expand reality who wear corresponding input and output technologies directly on their bodies. an exception here is spatial augmented reality [7], which is mostly based on projections and projection mapping, requires a correspondingly large amount of preparation time, is spatially limited, and most often not or only slightly interactive. this vision paper, therefore, proposes a new form of xr: societal xr (highlighted in green in figure 1). its goal is to make xr interactively usable and experienceable for really to everyone, by not having to immerse people in virtual worlds, but by bringing the virtual offers to them into the real world. this is made possible by cameras, tracking systems, spatial audio speakers, projectors, and displays of all kinds in the real environment. it also requires modern approaches to semantically modeling intelligent environments (such as [8]), interpreting user behavior, storing user profiles, generating user interfaces at run-time, automatically choosing, and applying xr patterns (see [9]), and many more. societal xr is intended as a public form of xr that can also be perceived by bystanders and passersby and not only by its respective users. yet it can serve individuals as well as public and societal benefits. for this purpose, it integrates aspects of spatial augmented reality, mixed reality, social xr, ambient intelligence, and ubiquitous computing [10], among others. 2 a societal xr story it was a beautiful saturday in may, with bright sunshine and pleasant temperatures, when claire left her apartment, closed the front door behind her, and headed to the park to call her friend emilie. the sidewalks and playgrounds on claire’s street were busy; apparently, the whole city wanted to enjoy the wonderful weather. the shops were open and invited potential customers to buy with projection shop windows and virtual mirrors. passers-by stood in front of almost every shop. some got carried away by the personalized ads, pausing to look at promotional holographs chosen for them based on their customer profiles. others stopped in front of clothing stores, pointed to items, and were then able to look and move in the virtual mirrors of the shop windows wearing exactly those items of clothing. the latest craze, however, was holographic fittings that allowed passers-by to point to one or more items of clothing, which were then projected onto their bodies using holographic projectors. so not only could the passers-by see themselves in these clothes, but also companions, relatives, and other people in the vicinity could. the passers-by could even walk a few meters in their holographic clothing until they left the range of the respective projector. claire was lucky to live on such a popular shopping district, which meant there were projectors installed all over the street. they were all connected to each other. while each store also had dedicated projectors so that only those projectors could project that store’s merchandise, advertising, and other content, the projectors commissioned and installed by the city administration were also networked, distributed across the city, and were therefore capable of holographic projections maintained almost seamlessly even over several kilometers. many residents used this function as a matter of course. just as claire was about to start walking, two joggers ran past her together: one really present, the other projected. of course, they both jogged, but not in the same place. the second jogger could be anywhere, on another continent or on the other side of the world. but they both jogged in straight lines, so one was projected next to the other. with the help of either headphones or spatial audio, they could hear each other. via microphones or smartphones that they wore to record their words, they could talk as if they were really jogging side by side. claire had tried it herself a few times, even though jogging was not one of her favorite pastimes. for the first half hour she had had trouble with traffic lights and obstacles, because whenever her running partner had to stop at traffic lights or obstacles that claire could not see, she also had to stop, next to her running partner, in the middle of the sidewalk. but that was fine. she got used to it on their first run together. paradigmplus (2022) 3:2 4 daniel görlich . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . after two minutes of leisurely strolling, claire arrived at the travel agency where she had already booked her last summer vacation. in her early twenties, just a few years ago, claire had never thought she would set foot in a business as old-fashioned as a travel agency; after all, she could have had her trips booked online. but since travel agencies had started offering virtual trips, virtual trip previews, and holographic travelogues, they had become more popular. some travel agencies had even set up or rented adjoining rooms in order to be able to offer customers the most convincing, interactive projections. they did not yet agree on the names of these rooms: some still used the old term cave, i.e., cave automatic virtual environment, while others preferred the term holo chamber. in any case, these rooms allowed their customers to be truly present in completely realistic-looking recordings or live transmissions from foreign locations. of course, the travel agency’s cameras had recognized and identified claire a few meters earlier. servers had then identified claire as a recurring customer, compared her customer profile at the travel agency, her advertising profiles on social media and the current summer trends. based on this, the algorithms had apparently decided on fiji. and so, the beautiful beaches of the coral coast on fiji’s main island viti levu were projected onto the virtual shop window, together with shots of the kula wild adventure park, the savu na mate laya waterfall and of course a fantastic hotel complex. claire watched enthusiastically and wished she had booked directly, but first she wanted to speak to a friend who was in the maldives at the time. many travel agencies now also offered this service in their cave’s or holo chambers: one could simply go live to a place where another person was vacationing and where the respective hotel or travel agency had cameras, microphones, thermometers, and projectors installed specifically for virtual calls. and so, after entering the travel agency and having a short conversation with today’s service employee, claire found herself back in the maldives. she talked to her friend, felt the summer heat, which of course was simulated by the installed air conditioning, and enjoyed a few minutes of downtime in one of the most popular holiday paradises in the world. this service was not exactly cheap, but claire was happy to pay a little more for a little break. it was definitely a good extra income for the travel agencies and hotels. after almost twenty minutes, claire said goodbye happily and contentedly and left the holo chamber. the service staff received her friendly. because claire still wanted to make some inquiries about her vacation idea, he led her to his desk in the service area and asked her about her preferred destination. cairo and giza, claire replied. both sat down. the employee typed briefly on his keyboard and after fractions of a second the floor of the travel agency turned into a desert. the walls showed a bright blue sky and the wall behind the employee, right in front of claire’s eyes, showed the pyramids of giza. the air conditioner raised the temperature by a few degrees, creating a light, warm draft. the overhead lighting was intensified accordingly. claire felt present right there, as if she really were in giza. she wondered, once again, how much a display film with a transparent protective layer would cost for her own apartment. only a few of her acquaintances had previously afforded this expensive luxury. apparently out of nowhere, a boy appeared next to the service employee. he was obviously a holographic projection. the employee excused himself with a smile and a gesture, turned to the boy, his son, who was on his lunch break at school, and spoke briefly with him about lunch and dinner and when to pick him up. then they said goodbye with one of those virtual hugs that seemed surprisingly real, even though they could not actually touch each other physically. the employee then turned back to claire, politely apologized, and spoke to her about the current cairo and giza travel deals. with a concrete idea, but not yet having booked, claire went back onto the street. sometimes she wondered why she should travel at all when she could also experience vacation completely virtually, but in fact these were two completely different things. it was actually a bit cooler back on the road, despite the sunny may day. everywhere, virtually projected people walked next to real people. they spoke to each other, but even at close range, claire could never hear their conversations because they spoke and heard each other through headphones, smartphones, or spatial audio. when a virtual person said goodbye, they simply disappeared. some people even walked virtual dogs. in big cities like new york, claire had heard, there would even be paradigmplus (2022) 3:2 societal xr a vision paper 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . extravagant locals who would keep virtual crocodiles and lions as pets on a leash. claire was glad that in her tranquil small town it was mostly just dogs. claire wanted to cross the street a few meters further, but just before she reached it, the pedestrian light switched to red. at the same time, virtual barriers rode up from the sidewalk, which in themselves were clear indications to pedestrians and cyclists—but as is so often the case, two pedestrians who were obviously in a particular hurry stepped through the barriers as if they were air, and quickly crossed the street. claire preferred to give herself a little more rest. after less than a minute, the barriers went back into the ground and in the split second in which they completely disappeared into it, the pedestrian traffic light switched to green. at the same time, barriers had moved up the street and blocked the lanes, at least virtually. now they formed an aisle, a closed path for pedestrians, in which they could supposedly safely cross the street. claire’s destination was the park just across the street. it was a small, idyllic park that families in particular used when the weather was good. every june, july, and september, the city held virtual festivals there. while it used to be pure projection mapping in the past, with buildings and trees bathed in bright colors, for a few years now it had been interactive simulations with annually changing themes. most of the time, however, the large lawns were available to visitors for picnics, frisbee games, and other recreational activities. stages were sometimes set up for plays and concerts, and many days street musicians performed live. this was also the case today: already on the other side of the street, at the entrance to the park, a single violinist played covers of well-known rock and pop songs together with a virtual cellist and an equally virtual contrabass player. claire stood for a while and listened to the unlikely trio. four passers-by had transferred donations to the violinist with quick swiping gestures on their smartphones without even stopping. after three songs, claire also picked up her smartphone, called up her mobile payment service and with a single wave of her right hand swiped two dollars towards the violinist, who smiled at her in thanks and continued playing undeterred. since her smartphone had only identified one active payee in the vicinity, the assignment had occurred automatically. when claire entered the park, she already saw numerous children releasing virtual balloons. within the field of view of the ubiquitous cameras, the children could display a selection of shapes and colors with a gesture, which was projected in front of them as a radial menu. the city had paid $350,000 to equip this park with a sufficient number of cameras, spatial audio speakers and projectors. the skytracker projectors had been most expensive, and could be used to project virtual balloons, gliders, dragons, or occasionally alien spaceships over the park even in daylight. they could still be seen from a kilometer away, albeit faintly and blurry. and, of course, the projectors tricked. the small zeppelin, which appeared to be hovering several kilometers high and advertised a nearby amusement park, was of course much smaller and projected much closer. a few steps further, claire approached a fountain that was shut down for repairs. it was hard to see from a distance because the fountain continued to splash, at least virtually, while lettering rotated around it announcing its repair and reopening for next friday. from time to time even fish jumped out of the water and plunged straight back in. as claire approached the fountain to get a closer look at the fish, a virtual barrier rose out of the ground directly in front of her, blocking her path. it was only then that claire noticed the holographic foil display that had been spanned around the construction site, completely blocking the view of the actual fountain. everything claire had seen was a deceptively real-looking simulation. as claire stood in front of the virtual barrier, she noticed a father and his five-year-old daughter. unintentionally, claire overheard her father calling her dorothy as she begged, "may i? may i?". at first, claire did not know what dorothy meant, but when her father finally gave in, the girl opened a radial menu with a gesture and deliberately selected a specific option. the menu then disappeared and the paved sidewalk beneath her feet turned into a yellow brick walkway like the one that led to the emerald city in the wizard of oz. the girl cheered happily, took her father by the hand, and pulled him along the path, further into the park. claire looked after them with a smile and was surprised that she had never noticed that there was also a display foil on the asphalt and a transparent protective paradigmplus (2022) 3:2 6 daniel görlich . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . layer on top of it. some things apparently only children knew! claire continued to stroll down the yellow brick path until she could no longer see dorothy and her father. instead, she discovered the sub-surface stage, which, exceptionally, was provided with real, physical barriers. behind these was basically just a large, nearly cubic hole in the ground, but equipped with numerous projectors. as was usually the case, it was used to represent the enclosure of a zoo from some other city in the world. this week there was a time-delayed live broadcast from the beijing zoo’s panda enclosure, according to the displayed virtual signage. due to the difference in time zones, claire saw a recording that had actually been recorded at the same local time in beijing, several hours earlier. everything looked deceptively real. families and couples stood behind the barrier and watched as three panda babies played with and besieged the zookeeper while he tried to clean up the enclosure. a real test of patience for him, but a huge spectacle for the children here in the park. as she watched the pandas, claire’s phone vibrated. since she wanted to make the call undisturbed, without too many other people around her, claire initially only accepted the audio call, but not the live projection call. claire apologized to her friend emilie because, while it was not considered rude these days, it was at least unusual to make a call without a live projection. but emilie took it easy, and after a few meters, which she had walked quickly, claire reached an unoccupied park bench. she sat down on one half of the bench and swiped the projection mode icon towards the other half. she knew that cameras were capturing her sitting on the bench and streaming it lifelike into emilie’s living room. emilie now also appeared lifelike next to her, sitting on the other half of the bench. she was probably sitting in a chair, an armchair or on a couch at home which made it easy for the algorithms programmed into the projection systems to position her so precisely on the bench that emilie even seemed to be leaning against the backrest. claire and emilie talked, laughed, and celebrated emilie’s ph.d. degree for over an hour. claire was looking forward to hopefully visiting her in paris next year. the eiffel tower was said to have some awesome societal xr specials in the program that claire definitely wanted to see live. 3 discussion the vision presented herein will not be feasible in the 2020s; that is not its goal either. although numerous presented technologies such as virtual mirrors [11] already exist, many of them are still too expensive, too vulnerable to weather and damage, too controversial in society, or not sufficiently regulated by law. for example, the vision in section 2 does not include any long-range audio recording, which is actually already commercially available via directional microphones such as shot-gun microphones and can be widely used. this possibility was intentionally left out because it seems unattractive for reasons of privacy and data protection, at least nowadays, even if this is only a weak justification, because phone calls or conversations can also be overheard, bugged, and recorded in plenty of other ways. nevertheless, it is likely that without significant changes in the legislation of numerous countries and in public opinion, no acceptance could be achieved within a social discourse for plastering the public space with microphones—although analogously the public areas in many cities such as london, despite considerable concerns, rejection and resistance, have long been comprehensively monitored with cameras and have been recorded and made visible to everybody from anywhere in the world, by google street view and the like. in this vision paper, too, the limits of what is technically feasible and what is socially desirable must therefore be discussed. for a vision of a socially acceptable, inclusive technology for the common good, it is not sufficient just to provide technologies, but legislative, legal, and societal preconditions must also be taken into account. for example, the vision outlined requires user profiles and the collection of user data on a scale that is already being pursued by social media and companies in the advertising industry, e.g., for personalized ads (for example, [12]), although many people refuse them for numerous reasons (cf. [13]). in order to fulfill the vision of a user-centered xr paradigm for the common good, which paradigmplus (2022) 3:2 societal xr a vision paper 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . benefits the general public instead of bringing profits to the few, the path to societal xr would therefore require a large number of regulations, from laws to voluntary commitments by the industries involved to user-friendly terms of service. the path to societal xr also requires rethinking in many other areas, particularly in the way user interfaces are developed. nowadays, they are mostly developed at design time, long before the completion of their respective software and its use. user interfaces usually do not adapt at all or only slightly to the respective user and are usually limited to a single device, e.g., the smartphone or tablet pc currently in use. they are also designed to be used by only one user at a time who is usually identified via a login procedure. however, approaches such as ubiquitous computing, ambient intelligence and also societal xr require shifting the generation of user interfaces from design time to run-time to allow for generating adaptive user interfaces on demand that reach across numerous devices, rooms or even environments. actually, the who environment may become the user interface. in the vision presented above, the user interface may even follow the user by jumping over from one device to the next one and may change significantly when another potential user arrives. the basics of run-time ui generation for intelligent environments have already been researched, e.g., in the context of the world’s first industry 4.0 factory [14]. it was found that they required extensive, semantically precise descriptions and models, e.g., of users and user tasks, of usage environments, and of automating ui design approaches such as the application of design patterns [15]. many of the technologies mentioned above are already available today: virtual mirrors, personalized ads, radial menus, ambient lighting, spatial audio, etc. others are currently in the process of gaining social acceptance, due, for example, to major virtual events such as the abba voyage concert in london in 2022. and some are still futuristic, but, thanks to science fiction, nevertheless already known to a broad public, such as holo chambers, called holodecks or holosuites in star trek, among many others. a few movies such as minority report (steven spielberg, 2002) feature technologies that are close to the societal xr vision presented above but focus on their potential dystopian elements. yet, the basic idea behind this vision is not a technical, but a social one. today’s virtual, augmented, and mixed reality paradigms, as well as xr in general, mostly focus on one or a few individuals who, either alone or in a small group, experience additional virtual content that everyone else around them does not perceive. this not only means immersion in a positive sense, for example immersion in a virtual world or a mixed reality, but also always means the exclusion of others, even nearby people. immersing oneself in a virtual reality with all senses always means disengaging oneself from reality for the most part at the same time. some studies such as aardema et al., [16] suggest that the use of virtual reality can cause dissociative experiences such as depersonalization and derealization as well as a lessened sense of presence in the objective reality. with increasing user numbers, one also has to ask the question how a future society should function in which millions of people regularly disengage from reality in order to immerse themselves in virtual worlds, just as in ready player one [17]. but xr can also achieve completely opposite goals, such as mass events in which any number of people and, above all, all people can take part, regardless of age, technical skills, or disabilities. this is made possible, for example, by spatial augmented reality or projection mapping (cf. [18]), not only on stages or in concert halls, but also outdoors, e.g., in a city center or park. approaches such as spatial augmented reality are currently hardly researched. technical development hardly takes place in those approaches, but focuses on head-mounted displays for vr, ar, and mr, and virtual worlds and spaces. it would be worthwhile, in addition to the development of virtual spaces that are decoupled from reality, to also research the supplementation of reality or the real environment with meaningful, generally accessible and socially relevant virtual content. the vision of societal xr can provide a common concept for this. ultimately, however, societal xr, like any technological vision, is limited and will ultimately be overtaken by the future—which is exactly how it has to be. this results in particular from the constantly falling costs for hardware and the democratization of technologies, as historically been the case with the pc and the tablet pc: between 1975 and 1980 (the dates vary) microsoft’s founders paradigmplus (2022) 3:2 8 daniel görlich . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . paul allen and bill gates envisioned "a (micro)computer in every home and on every desk", which has long since been overtaken by reality. in [10], as early as 1991, mark weiser wrote about three types of "ubiquitous computers ... in different sizes, each suited to a particular task": tabs, pads, and boards. all three device types are available by now, and "weave themselves into the fabric of everyday life" [10], but there are already significantly more device types such as wearable computers for many more use cases. every technological vision will ultimately, in one way or another, be overtaken by reality, which is a positive sign because a vision must not be utopian and impossible to implement but should rather offer a guide for research and development in the coming years or decades. the vision presented herein could have easily turned out more utopian, featuring, for example, complete immersion through psychochemistry as in stanislaw lem’s "the futurological congress" (1971), totally natural haptic feedback, holodecks as in "star trek," or ubiquitous speech recording and recognition devices that, as discussed above, could be installed everywhere in the environment instead of being worn on the body, just like today’s surveillance cameras. a vision of societal xr which worked completely without any body-worn devices, would have been easy to formulate. however, in order to serve as a guide for xr research and development, a goal was needed that appears both socially desirable and technologically feasible within a medium-term period. because the societal xr vision presented in this paper is limited, it cannot solve all current and future problems. for example, the digital divide that already exists today may be reduced if or when xr becomes freely available to everyone in the public space, so that individuals will not have to buy expensive devices such as mixed reality hmd’s that they may not be able to afford. the fact that reducing the digital divide is not equivalent to abolishing it is briefly addressed in section 2: at one point the vision describes that claire wonders how much a technology like that in the travel agency might cost, because only few of her acquaintances have installed something similar in their homes. this was not deepened in this discussion because the focus of the societal xr vision is on public, freely accessible xr. however, a similar vision for individual needs, e.g., at home, would also be desirable. 4 conclusion societal xr is a vision that challenges three core aspects of current xr research, development, and technologies: it presents a concept of a generally accessible, non-discriminatory, social, and public form of reality expansion that excludes nobody. in societal xr, virtual elements become part of the perceived reality. they come to the users instead of the other way around. they do not require the wearing of smart glasses or head-mounted displays and do not create a limited virtual world or space for a limited number of participants. societal xr combines concepts such as spatial augmented reality, social xr, mixed reality, ambient intelligence, and ubiquitous computing to formulate a new vision of what can be achieved with xr when no longer each and every individual has to afford xr and its devices, but society as a whole decides to integrate virtuality much more into reality and everyday life than is usual today. it is intended as a utopian rather than a dystopian concept of an evolutionary further development of xr that breaks away from xr being used individually or in small groups, but instead making it publicly available to entire towns, societies and populations. although the societal xr vision cannot be adopted instantaneously, but might become feasible only within a few decades, it is to be hoped that it will be adopted and thereby give xr research and development an additional new direction and goal. authors’ information – daniel görlich studied informatics at humbolt university of berlin and holds a ph.d. in humanmachine interaction from the university of kaiserslautern. currently, he is a professor for virparadigmplus (2022) 3:2 societal xr a vision paper 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . tual reality and game development at srh university heidelberg. his research focuses on modern and novel ways to design and develop virtual worlds. authors’ contributions – daniel görlich is the sole author. competing interests the authors declare that they have no competing interests. funding no funding was received for this project. references [1] m. ipe, “virtual worlds: new media for communication,” rajagiri management journal, vol. 5, no. 2, pp. 37–42, 2009. available at http://journals.rajagiri.edu/index.php/rmj/article/ view/258. [2] t. bezmalinovic, “the tinyverse could beat the metaverse – at least for starters.” available at https://mixed-news.com/en/ the-tinyverse-could-beat-the-metaverse-at-least-for-starters/, 2022. [3] l.-h. lee, t. braud, p. zhou, l. wang, d. xu, z. lin, a. kumar, c. bermejo, and p. hui, “all one needs to know about metaverse: a complete survey on technological singularity, virtual ecosystem, and research agenda,” arxiv preprint arxiv:2110.05352, 2021. available at https: //doi.org/10.48550/arxiv.2110.05352. [4] j. mcveigh-schultz, a. osborne, m. kreminski, s. fernandes, s. fielder, v. li, t. lang, and k. isbister, “social superpowers in social vr. beyond approximation of face-to-face,” in social vr workshop in 2021 acm chi virtual conference on human factors in computing systems, pp. 1–5, 2021. [5] g. freeman and d. maloney, “body, avatar, and me: the presentation and perception of self in social virtual reality,” in proceedings of the acm on human-computer interaction, pp. 1–27, 2021. available at https://doi.org/10.1145/3432938. [6] a. s. praetorius and d. görlich, “how avatars influence user behavior: a review on the proteus effect in virtual environments and video games,” in international conference on the foundations of digital games, pp. 1–9, 2020. available at https://doi.org/10.1145/3402942.3403019. [7] o. bimber and r. raskar, spatial augmented reality: merging real and virtual worlds. crc press, 2005. [8] d. görlich and k. breiner, “useware modeling for ambient intelligent production environments,” in proceedings of the models’07 workshop on model driven development of advanced user interfaces, 2007. available at http://ceur-ws.org/vol-297/paper2.pdf. [9] d. görlich, t. akincir, and g. meixner, “an overview of user interface and interaction design patterns for vr, ar, and mr applications,” in proceedings of the augmenting usability evaluation workshop at mensch und computer, 2022. paradigmplus (2022) 3:2 http://journals.rajagiri.edu/index.php/rmj/article/view/258 http://journals.rajagiri.edu/index.php/rmj/article/view/258 https://mixed-news.com/en/the-tinyverse-could-beat-the-metaverse-at-least-for-starters/ https://mixed-news.com/en/the-tinyverse-could-beat-the-metaverse-at-least-for-starters/ https://doi.org/10.48550/arxiv.2110.05352 https://doi.org/10.48550/arxiv.2110.05352 https://doi.org/10.1145/3432938 https://doi.org/10.1145/3402942.3403019 http://ceur-ws.org/vol-297/paper2.pdf 10 daniel görlich . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [10] m. weiser, “the computer for the 21st century,” acm sigmobile mobile computing and communications review, vol. 3, no. 3, pp. 3–11, 1999. available at https://doi.org/10.1145/329124. 329126. [11] c. bichlmeier, s. m. heining, m. feuerstein, and n. navab, “the virtual mirror: a new interaction paradigm for augmented reality environments,” ieee transactions on medical imaging, vol. 28, no. 9, pp. 1498–1510, 2009. available at https://doi.org/10.1109/tmi.2009.2018622. [12] t. p. tran, “personalized ads on facebook: an effective marketing tool for online marketers,” journal of retailing and consumer services, vol. 39, pp. 230–242, 2017. available at https://doi. org/10.1016/j.jretconser.2017.06.010. [13] d. dobrinić, i. gregurec, and d. dobrinić, “examining the factors of influence on avoiding personalized ads on facebook,” zbornik radova ekonomskog fakulteta u rijeci: časopis za ekonomsku teoriju i praksu, vol. 39, no. 2, pp. 401–428, 2021. available at https://doi.org/10.18045/ zbefri.2021.2.401. [14] k. breiner, d. görlich, o. maschino, g. meixner, and d. zühlke, “run-time adaptation of a universal user interface for ambient intelligent production environments,” in international conference on human-computer interaction, pp. 663–672, 2009. available at https://doi.org/10.1007/ 978-3-642-02583-9_72. [15] d. görlich and k. breiner, “intelligent task-oriented user interfaces in production environments,” ifac proceedings volumes, vol. 40, no. 16, pp. 471–476, 2007. available at https: //doi.org/10.3182/20070904-3-kr-2922.00084. [16] f. aardema, k. o’connor, s. côté, and a. taillon, “virtual reality induces dissociation and lowers sense of presence in objective reality,” cyberpsychology, behavior, and social networking, vol. 13, no. 4, pp. 429–435, 2010. available at https://doi.org/10.1089/cyber.2009.0164. [17] s. spielberg, a. silvestri, z. penn, e. cline, and d. de line, ready player one. warner bros usa, 2018. [18] t. burch, “move aside ar & vr. projection mapping is the next retail reality.” available at https: //talk-retail.co.uk/move-aside-ar-vr-projection-mapping-next-retail-reality/, 2016. paradigmplus (2022) 3:2 https://doi.org/10.1145/329124.329126 https://doi.org/10.1145/329124.329126 https://doi.org/10.1109/tmi.2009.2018622 https://doi.org/10.1016/j.jretconser.2017.06.010 https://doi.org/10.1016/j.jretconser.2017.06.010 https://doi.org/10.18045/zbefri.2021.2.401 https://doi.org/10.18045/zbefri.2021.2.401 https://doi.org/10.1007/978-3-642-02583-9_72 https://doi.org/10.1007/978-3-642-02583-9_72 https://doi.org/10.3182/20070904-3-kr-2922.00084 https://doi.org/10.3182/20070904-3-kr-2922.00084 https://doi.org/10.1089/cyber.2009.0164 https://talk-retail.co.uk/move-aside-ar-vr-projection-mapping-next-retail-reality/ https://talk-retail.co.uk/move-aside-ar-vr-projection-mapping-next-retail-reality/ introduction a societal xr story discussion conclusion vol. 3, no. 3, pp 16 29, 2022 doi: 10.55969/paradigmplus.v3n3a2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . hybrid approach for phishing website detection using classification algorithms mukta mithra raj1 � and j. angel arul jothi1 1birla institute of technology and science pilani, united arab emirates f20190134@dubai.bits-pilani.ac.in, angeljothi@dubai.bits-pilani.ac.in abstract the internet has significantly altered how we work and interact with one another. statistics show 63.1% of the present world population are internet users. this clearly indicates how heavily man is dependent on digital media. digital media users are on the rise and so is the incidence of cybercrimes. people who lack experience and knowledge are more vulnerable and susceptible to phishing scams. the victims experience severe consequences as their personal credentials are at stake. phishers use publicly available sources to acquire details about the victim’s professional and personal history. countermeasures must be implemented with the highest priority. detection of malicious websites can significantly reduce the risk of phishing attempts. in this research, a highly accurate website phishing detection method based on url features is proposed. we investigated eight existing machine learning classification techniques for this, including extreme gradient boosting (xgboost), random forest (rf), adaptive boosting (adaboost), decision trees (dt), k-nearest neighbors (knn), support vector machines (svm), logistic regression and naive bayes (nb) to detect malicious websites. the results show that xgboost had the best accuracy with a score of 96.71%, followed by random forest and adaboost.we further experimented with various combinations of the top three classifiers and observed that xgboost-random forest hybrid algorithms produced the best results. the hybrid model classified the websites as legitimate or phishing with an accuracy of 97.07% keywords: url features ⋅ data mining ⋅ machine learning ⋅ hybrid classification algorithms ⋅ phishing website detection received: 27 october 2022 ⋅ accepted: 16 december 2022 ⋅ published: 20 december 2022. 1 introduction website phishing [1] is the unethical practice of designing websites that appear genuine and legitimate but are actually created to gain unauthorized access and steal information. a malicious website attempts to deceive people into revealing private information. it accomplishes this by making you believe that the website is genuine. for many people, the internet serves as a window to the outside world. society’s reliance on the digital world has increased during the pandemic. they use it for a variety of tasks, including storing data and conducting online financial transactions. consequently, there was an increase in phishing websites and many people fell prey to phishing attacks. unknowingly, the vulnerable customers would attempt to click on the fake urls that appeared cloned and genuine. they are stripped of their personal information and login details by such harmful phishing this article is distributed under the terms of the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. https://doi.org/10.55969/paradigmplus.v3n3a2 hybrid approach for phishing website detection using classification algorithms 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . attacks. customers and businesses are both at risk, which might result in major financial loss and business interruption. eighty-five percent of phishing attacks are aimed at stealing user credentials, according to ibm’s 2021 [2] data breach report. this will result in additional intrusions and cybercriminals can access files and observe user behavior. there could be severe and even severe consequences for a business, its staff, and its clients. prevention is the key to altering disastrous consequences. a cutting-edge automatic phishing detection system is required to counter dangerous threats. models incorporated into web browsers can identify phishing activities. automated phishing detection algorithms can become more efficient if website properties are integrated with the input dataset from a large number of websites. machine learning classification algorithms can detect phishing websites efficiently in less time. compared to the deep learning method, it works effectively with less training data and is much faster. another advantage of machine learning over deep learning is that it does not require specific hardware like graphics processing units (gpu) for implementation. therefore, the objective of this research is to apply machine learning classification algorithms for the accurate identification of phishing websites. eight different classification algorithms are used and tested on the dataset, including extreme gradient boosting (xgboost), random forest (rf), adaptive boosting (adaboost), decision trees (dt), k-nearest neighbors (knn), support vector machines (svm), logistic regression and naive bayes (nb) to identify malicious urls. the results are compared, and the best three in terms of accuracy are utilized to construct hybrid algorithms. the research also explores the impact of using fewer features on the classification performance of the classifiers. the research report is structured as follows: related work is included in section 2. the dataset is described in section 3 of the document. section 4 details the approach and the classification algorithms. implementation details are given in section 5. the findings and discussion are presented in section 6. section 7 concludes the research paper with a scope of future work. 2 related works this section includes a brief overview of the ensemble approaches presented in some of the recent works. an ensemble model was presented by jiaqi gu and hui xu [3]. the proposed model is developed by integrating xgboost with knn and rf as base learners. when compared with other ensemble models, the xgboost ensemble model achieved better run time and an accuracy of 96.44%. further, it can safeguard internet users when embedded into an application or web extension. in the proposed work by maini et al. [4] url features were tested using eight classification algorithms, namely, xgboost, rf, adaboost, dt, knn, svm, logistic regression, and nb. xgboost attained the highest accuracy of 93.2%. the voting classifier is then used to construct the ensemble model. according to the results, the suggested ensemble technique performed better than xgboost, with an accuracy of 93.6%. an efficient random forest-support vector machine (rf-svm) hybrid model was developed by pandey et al. [5]. according to the experimental results, the accuracy of the rf-svm hybrid model is superior to that of the rf and svm. the dataset is divided into multiple parts and trained using random forest. each subunit is bagged to create the final decision unit. to improve accuracy, the rf subunits are then categorized using svm. the svm and rf results are bagged to merge the results. the hybrid model can detect phishing websites with a high rate of accuracy. in a similar study, ramana et al. [6] developed an ensemble of xgboost, rf, and dt based on feature selection to classify phishing websites. experiments were conducted using uci and mendeley datasets. the trial with the former achieved an accuracy of 97.51% while that with the latter scored 98.45%. the final model was implemented using wrapper feature selection techniques and filters. paradigmplus (2022) 3:3 18 mukta mithra raj and j. angel arul jothi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . the efficiency of ensemble models in detecting phishing websites is also evident in the svm, decision tree, and random forest ensemble model proposed by abusaimeh et al. [7]. the proposed model was compared with each classifier and thereafter implemented and analyzed using the dataset. the results demonstrate that the ensemble model outperforms the three algorithms. the hybrid framework developed by tabassum et al. [8] based on merging bagging and boosting techniques was built using svm, dt, rf, and xgboost. the url features were selected and to achieve better results, the dimensionality of the feature subset was reduced. an accuracy of 98.28% was obtained using the hybrid approach. another ensemble method was devised by lakshmanarao et al. [9] by combining rf, dt, and gradient boosting. a variety of classification techniques were tested utilizing the dataset after preprocessing. later, two sets of priority algorithms were created grouping the individual classifiers. the best-performing algorithms from the two sets were combined to construct the final ensemble algorithm. the proposed model achieved an accuracy of 97%. the ensemble framework suggested by subasi et al. [10] consists of svm and adaboost classification algorithms for the detection of phishing websites. the approach improved accuracy, f1 score, and roc curve. according to the results obtained, the ensemble model can be used to detect phishing pages with a success rate of 97.61 percent. zamir et al, [11] proposed two stacking models. the first model combines bagging, neural networks, and random forests. the second model consists of random forest, knn, and bagging. principal component analysis (pca) was used to increase the proposed models’ accuracy. the first stacking model achieved the highest accuracy of 97.4% when compared to the individual classifiers and the second stacking model. kalabarige et al. [12] proposed a multilayered stacked ensemble learning technique (mlselm ) using the multi-layer perceptron (mlp), xgb, knn, rf, and lr at various layers to improve performance. four distinct datasets were used to evaluate the model, and the accuracy ranged from 96.79% to 98.90%. the proposed model performed better with a balanced dataset than with an imbalanced dataset. the machine learning approach formulated by makkar et al. [13] evaluated bagged adaboost, bayesian generalized linear model, naïve bayes, linear svm with class weights, ensembles of generalized linear models, monotone multilayer perceptron neural network, quadratic discriminant analysis, multilayer perceptron, neural networks with feature extraction, and oblique rf. following that, 10 rounds of cross-validation are performed on an ensemble of the best three best in terms of accuracy. a total accuracy of 97.27% is shown by the tested models. yang et al. [14] proposed an ensemble method using rf and convolutional neural networks (cnn) to predict the authenticity of websites. the integration of cnn improved the performance of the model. urls are transformed into fixed-size matrices using character embedding strategies. the proposed model recorded a 99.35% accuracy rate. in the study by al-sarem et al. [15] an improved stacking ensemble technique for phishing website detection was proposed. random forests, adaboost, xgboost, bagging, gradientboost, and lightgbm were among the ensemble machine learning techniques whose parameters were optimized using a genetic algorithm (ga). the top three models were used to build the ensemble classifiers. according to the experimental findings, the proposed optimal stacking ensemble method achieved an accuracy of 97.16%. a voting ensemble model, an expandable random gradient stacked voting classifier ( erg-svc) was proposed by indrasiri et al. [16] after experimenting with seven classifiers. 22 best features were selected after feature selection and the trials were conducted with different datasets. with a prediction accuracy of 98.118%, gb surpassed other algorithms individually while the ensemble model achieved an accuracy of 98.27%. alsaedi et al. [17] proposed an ensemble model with rf and mlp. rf is used for pre-classification while mlp is used for decision-making. to enhance detection performance, features based on cyber threat intelligence (cti) are applied. compared to the conventional url-based model, the proposed paradigmplus (2022) 3:3 hybrid approach for phishing website detection using classification algorithms 19 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . cti-based detection model has an improved accuracy of 7.8%. the goal of this study is to recommend a hybrid classification model. thus, in our work, we analyzed and considered eight different machine learning classifiers applied in the publications. the accuracy rate and f1 score are measured using 30, 15, 10, and 5 url features. 3 dataset description the kaggle1 dataset for phishing websites is utilized in this work. it includes the urls of more than 11,000 websites. each url has 30 attributes. values 1 and -1 are used to classify the websites as legitimate and phishing respectively. table 1 lists the parameters of a url, its types, and its values. values range from two to three which represent an attribute’s range of strength from low to high. table 1: url attributes, types, and values attribute attribute type value usingip, shorturl, symbol@, redirecting, prefixsuffix, domainreglen, favicon, nonstdport, httpsdomainurl, requesturl, infoemail, abnormalurl, statusbarcust, disablerightclick, usingpopupwindow, iframeredirection, ageofdomain, dnsrecording, pagerank, googleindex, statsreport categorical {-1,1} longurl, subdomains, https, anchorurl, linksinscripttags, serverformhandler, websitetraffic, linkspointingtopage categorical {-1,0,1} websiteforwarding categorical {0,1} 4 methodology phishing websites can be identified by analyzing the url features of the website. figure 1 illustrates the method used in this work to classify the websites. definite features that constitute each url are selected. the selected features are then utilized for training and testing in which 70% of the samples in the dataset are used for training and 30% for testing. the classifiers and hybrid models are then trained to determine whether a website is phishing or not. 1https://www.kaggle.com/code/eswarchandt/website-phishing/data figure 1: proposed architecture. paradigmplus (2022) 3:3 https://www.kaggle.com/code/eswarchandt/website-phishing/data 20 mukta mithra raj and j. angel arul jothi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . the algorithms applied for the classification of the websites are extreme gradient boosting (xgboost), random forest, support vector machine, naive bayes, logistic regression, k-nearest neighbours, decision tree, and adaboost. the following subsections explain these classification algorithms. 4.1 machine learning algorithms 4.1.1 xgboost xgboost is typically used for regression and classification on large datasets. using sequentially built shallow decision trees and a highly scalable training method that reduces overfitting, it produces accurate results. the learning rate, number of trees, percentage of randomly sampled columns, gamma, and maximum depth per tree are among the significant hyperparameters. as performance improves, complexity and overfitting also increase. overfitting is enhanced by the column fraction. regularization depends on the value of the regularization parameter gamma, which increases as the gamma value increases [18]. 4.1.2 random forest (rf) the rf classifier algorithm uses a collection of distinct decision trees that have not been pruned and are fully developed. this algorithm uses randomness to generate each of the separate trees, which are then combined to generate a prediction. for large datasets, this approach is more effective than a single decision tree and creates far less variance. it almost exactly approximates missing data. the number of trees, the minimum number of data objects required to divide an internal node, the minimum number of samples needed to be present in the leaf node, and the number of jobs that run simultaneously are the hyperparameters used [19]. 4.1.3 support vector machine (svm) the svm algorithm can be applied to classification or regression tasks. the key component of it is a hyperplane that splits the attribute space into two groups: legitimate or phished websites. the classification of the websites is accomplished by placing the legitimate websites on one side of the plane and the phished websites on the other once the algorithm has been trained using the dataset. potential errors caused by overtraining are lessened as a result. the kernel hyperparameter and the probability hyperparameter specify the type of algorithm to be used to enable the probability estimates [20]. 4.1.4 naive bayes (nb) the nb classification algorithm gives predictions based on the probability of an object. in this study, an nb classifier called bernoulli naive bayes was used, and the features comprised boolean variables. features are regarded as independent of one another. the only parameter that cannot be modified is the number of classes [20]. 4.1.5 logistic regression logistic regression is used for both classification and regression. it is widely used for binary classification problems. lr uses the sigmoid function to determine a label’s probability [20]. 4.1.6 k-nearest neighbors (knn) the knn approach is primarily employed for classification although it can be utilized for regression issues. it categorizes two data objects using a similarity or distance metric. k represents the number of neighbors and the value of k in this work is 5. among the crucial hyperparameters are the number paradigmplus (2022) 3:3 hybrid approach for phishing website detection using classification algorithms 21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . of neighbors, the distance metrics used to determine the neighborhood’s composition, and the weight function for prediction [18]. 4.1.7 decision tree (dt) dt constructs a tree structure to address classification and regression problems. the attribute with the highest information gain is taken as the splitting attribute or the decision node. the class label is represented by the leaf nodes in the decision tree. the stopping criterion of a decision tree is when the child node has a homogeneous class. the number of trees is the most important hyperparameter used [18]. 4.1.8 adaboost adaboost, an ensemble technique, can boost the performance of any machine learning algorithm. it works best when employed with weak learners. with a learning rate of 1, the adaboost used here has 50 decision trees. selecting a weak learner to train the model, the number of weak learners to train, and the weights of the weak learners that affect the learning rate are the essential hyperparameters [10]. 4.2 hybrid model a hybrid model is constructed to improve the accuracy of the classification model. the voting classifier is used to create the hybrid classifiers. it is a machine learning model that is used to train an ensemble of algorithms to carry out classification tasks. hybrid models of xgboost and rf, xgboost and adaboost, and rf and adaboost are created using the voting classifier. the hybrid models are then trained using the same dataset, and the xgboost-rf hybrid classification model achieved the highest testing accuracy of 97.07%. 5 implementation this work was developed using the jupyter notebook accessed from anaconda with python version 3.10.4. the in-built machine learning library scikit-learn was utilized to build the eight classification models. the models were built by using 70% of the dataset for training and the rest 30% for testing. the confusion matrix is formed by considering the samples that fall into the predicted and actual classes and is used to formulate the evaluation metrics. the evaluation metrics used to compare the algorithms are accuracy, error rate, recall, specificity, precision, and f1 score given in table 2. these metrics are calculated using true positive (tp), false negative (fn), false positive (fp), and true negative (tn). when a legitimate website is correctly predicted as legitimate it is tp, whereas if the prediction is false, then it is fn. when a phishing website is rightly identified as phishing, it is tn, while if it is incorrect, it is fp. table 2: evaluation metrics true positive rate (tpr) / recall /sensitivity tp / (tp+fn) true negative rate (tnr) /specificity tn / (fp+tn) accuracy (tp+tn) / (tp+fp+tn+fn) error rate (1 accuracy) precision tp/ (tp+fp) f1 score 2 * precision * recall/ (precision + recall) paradigmplus (2022) 3:3 22 mukta mithra raj and j. angel arul jothi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 results and discussion an imbalanced dataset was used for training and testing the individual classifiers and the hybrid models. the significance of feature selection on the classification models was explored using different numbers of attributes. the performance of all models was initially examined using each of the 30 attributes, with the exception of the "index" feature because it is not used to classify websites. the experiment was carried out again with the top 15, 10, and 5 attributes for the models displayed in table 3. the average impurity decrease estimated from each decision tree was used for feature selection in adaboost, random forest, decision tree, k-nearest neighbors, xgboost, xgboost-random forest, xgboost-adaboost, and random forest-adaboost [21]. in the same way, the dataset attributes were assigned coefficients to determine the feature selection in logistic regression and svm. in naive bayes, the predictive value of a feature is evaluated by observing how the prediction error increases when a feature is missing. table 3: attributes used algorithms 15 attributes 10 attributes 5 attributes adaboost, decision tree https, anchorurl, websitetraffic, linksinscripttags, subdomains, prefixsuffix, serverformhandler, linkspointigtopage, pagerank, requessturl, googleindex, domainreglen, ageofdomain, usingip, longurl https, anchorurl, websitetraffic, linksinscripttags, subdomains, prefixsuffix, serverformhandler, linkspointigtopage, pagerank, requessturl https, anchorurl, websitetraffic, linksinscripttags, subdomains knn https, anchorurl, websitetraffic, subdomains, linksinscripttags, prefixsuffix, requesturl, serverformhandler, linkspointingtopage, ageofdomain, domainreglen, usingip, googleindex, dnsrecording, pagerank https, anchorurl, websitetraffic, subdomains, linksinscripttags, prefixsuffix, requesturl, serverformhandler, linkspointingtopage, ageofdomain https, anchorurl, websitetraffic, subdomains, linksinscripttags paradigmplus (2022) 3:3 hybrid approach for phishing website detection using classification algorithms 23 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . logistic regression anchorurl, prefixsuffix, https, websitefowarding, serverformhandler, linkspointingtopage, linksinscripttags, googleindex, subdomains, websitetraffic, usingip, nonstdport, httpsdomainurl, shorturl, dnsrecording anchorurl, prefixsuffix-, https, websitefowarding, serverformhandler, linkspointingtopage, linksinscripttags, googleindex, subdomains, websitetraffic anchorurl, prefixsuffix, https, websitefowarding, serverformhandler naive bayes https, prefixsuffix, anchorurl, subdomains, websitetraffic, usingip, googleindex, dnsrecording, ageofdomain, pagerank, requesturl, serverformhandler, linkspointingtopage, httpsdomainurl, websiteforwarding https, prefixsuffix, anchorurl, subdomains, websitetraffic, usingip, googleindex, dnsrecording, ageofdomain, pagerank https, prefixsuffix, anchorurl, subdomains, websitetraffic random forest, rf-adaboost https, anchorurl, websitetraffic, subdomains, prefixsuffix, linksinscripttags, requesturl, serverformhandler, linkspointingtopage, domainreglen, ageofdomain, usingip, dnsrecording, googleindex, pagerank https, anchorurl, websitetraffic, subdomains, prefixsuffix, linksinscripttags, requesturl, serverformhandler, linkspointingtopage, domainreglen https, anchorurl, websitetraffic, subdomains, prefixsuffix paradigmplus (2022) 3:3 24 mukta mithra raj and j. angel arul jothi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . svm anchorurl, prefixsuffix, https, websitefowarding, linkspointingtopage, linksinscripttags, serverformhandler, subdomains, googleindex, usingip, shorturl, nonstdport, websitetraffic, infoemail, httpsdomainurl, anchorurl, prefixsuffix, https, websitefowarding, linkspointingtopage, linksinscripttags, serverformhandler, subdomains, googleindex, usingip anchorurl, prefixsuffix, https, websitefowarding, linkspointingtopage xgboost, xgboost-rf, xgboost-adaboost https, anchorurl, prefixsuffix, serverformhandler, websitetraffic, linksinscripttags, subdomains, googleindex, shorturl, domainreglen, pagerank, linkspointingtopage, ageofdomain, usingip, infoemail https, anchorurl, prefixsuffix, serverformhandler, websitetraffic, linksinscripttags, subdomains, googleindex, shorturl, domainreglen https, anchorurl, prefixsuffix, serverformhandler, websitetraffic the performance of the classifiers for the various evaluation metrics for the attributes 30, 15, 10, and 5 are compared in figures 2, 3, 4, and 5 respectively. the performance of rf was optimized by determining the ideal value for the number of decision trees created. it was achieved by determining the average predictions and estimating the number of data items predicted for a specific dataset sample accurately, where the set of random values ranges from 40 to 300. the number of decision trees that worked best for 30 attributes was 200, followed by 20 for 15 attributes, 280 for 10 attributes, and 80 for the five most crucial attributes. from the results of figure 2 for 30 attributes, xgboost-random forest is the best classifier because it has the highest accuracy (97.07%) and f1 score (96.67%) among the classifiers. due to its low accuracy and poor f1 score, the naive bayes classifier exhibits the least performance. when compared to xgboost-adaboost, xgboost-rf displays good accuracy and f1 score but a lower precision value. as compared to xgboost-random forest, other classification models are found to have lesser accuracy, f1 scores, and recall values. for the 15 attributes shown in figure 3, random forest-adaboost has the highest accuracy (95.7%) and f1 score (95.44%), whereas the naive bayes classifier has the lowest. according to figure 4 for 10 features, xgboost-random forest and xgboost-adaboost are at the top of the list with accuracy rates of 94.96% and f1 scores of 94.31 percent, respectively, while naive bayes performs the worst. paradigmplus (2022) 3:3 hybrid approach for phishing website detection using classification algorithms 25 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 2: comparison of evaluation metric values for 30 attributes. figure 3: comparison of evaluation metric values for 15 attributes. adaboost has the best accuracy of 93.03%, while rf-adaboost has the highest f1 score of 92.11% for the five attributes represented in figure 5. compared to the other models, svm ranks the lowest. the results from tables 3 showed that xgboost-random forest performed the best. additionally, the performance of the classification was not enhanced by feature selection. the efficiency of the classifiers increases with more attributes and decreases substantially as the number of attributes is decreased. the dataset used does not contain any noise or irrelevant data and hence all attributes are paradigmplus (2022) 3:3 26 mukta mithra raj and j. angel arul jothi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 4: comparison of evaluation metric values for 10 attributes. figure 5: comparison of evaluation metric values for 5 attributes. considered to obtain a good result. the quality of the attributes in the dataset is more important than their quantity in order to get better results. the comparison of findings from this research and previous studies [3, 9] are given in table 4. [3] uses the same dataset as this study. an ensemble model comprising xgboost, rf, and knn is proposed in [3]. the outcomes of [3] demonstrate that the ensemble model outscored all other classifiers, attaining an accuracy and f1 score of 96.43% . paradigmplus (2022) 3:3 hybrid approach for phishing website detection using classification algorithms 27 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . similar datasets are used in this research and [9], except for one record. [9] utilized a dataset2 that comprised 11,055 records, whereas the current study utilizes 11,054 records. the best performance was produced by [9] using a stacking classifier made of rf, nn, and bagging, which had a 97.4% accuracy rate. moreover, pca was applied to reduce the features before classification. it can be seen from table 4 that [9] obtained an accuracy of 97.4% and f1 score of 97% while the proposed work is able to attain an accuracy of 97.07% and f1 score of 96.67%. the better performance of the model reported in [9] may be due to the use of a stacking classifier along with pca. the classifiers in this work are trained using an imbalanced dataset. however, the accuracy of such models can be deceptive. to overcome this issue, the dataset used in this research can be improved using sampling techniques. table 4: comparison with previous works method / evaluation metrics [3] [9] proposed (xgboost-random forest) accuracy 0.9643 0.974 0.9707 f1 score 0.9643 0.97 0.9667 7 conclusion phishing attacks are the most frequent attacks that impact organizations globally. many potential consequences may result from a successful phishing attempt. as a result of the breach, clients may leave the business causing huge financial losses. if an individual is targeted, their identity, credit, or financial information may be seriously compromised. uprooting this cybercrime is an uphill task. it is vital to develop and implement contingency measures to protect against such breaches promptly. users can be protected from malicious cyberattacks by identification and management of phishing websites. consequently, this research analyzes machine learning-based techniques for detecting phishing websites. eight classification algorithms comprising naive bayes, logistic regression, svm, k-nearest neighbours, decision tree, adaboost, random forest, and xgboost as well as the three hybrid models namely, rf-adaboost, xgboost-adaboost, and xgboost-rf are implemented and compared. we evaluated the work using an imbalanced dataset, with a higher percentage of legitimate urls than phishing urls. this allowed us to evaluate the security offered in a realistic setting because of the higher probability of legitimate urls in a practical scenario. the dataset is used to train and test each classification model and hybrid model. the classification technique using xgboost-random forest yields the best accuracy, with a score of 97.07%. it can therefore be used to identify phishing websites more precisely. the development of ensemble models utilizing various robust machine learning algorithms and deep learning techniques will be the focus of future research. authors’ information – mukta mithra raj is b.e. in computer science from the birla institute of technology and science pilani, dubai campus, united arab emirates. – j. angel arul jothi is assistant professor at the department of computer science of the birla institute of technology and science pilani, dubai campus, united arab emirates. 2https://www.kaggle.com/datasets/akashkr/phishing-website-dataset paradigmplus (2022) 3:3 https://www.kaggle.com/datasets/akashkr/phishing-website-dataset 28 mukta mithra raj and j. angel arul jothi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . authors’ contributions – mukta mithra raj participated in devising the concept of the article and in analyzing and interpreting the data for the research. – j. angel arul jothi participated in revising critically the article and approved the final version to be published. competing interests the authors declare that they have no competing interests. funding no funding was received for this project. references [1] j. fruhlinger, “what is phishing? examples, types, and techniques.” https://www.csoonline. com/article/2117843/what-is-phishing-examples-types-and-techniques.html, 2022. [2] ibm, “cost of a data breach report 2021.” https://www.dataendure.com/wp-content/ uploads/2021_cost_of_a_data_breach_-2.pdf, 2021. [3] j. gu and h. xu, “an ensemble method for phishing websites detection based on xgboost,” in 2022 14th international conference on computer research and development (iccrd), pp. 214–219, ieee, 2022. https://doi.org/10.1109/iccrd54409.2022.9730579. [4] a. maini, n. kakwani, b. ranjitha, m. shreya, and r. bharathi, “improving the performance of semantic-based phishing detection system through ensemble learning method,” in 2021 ieee mysore sub section international conference (mysurucon), pp. 463–469, ieee, 2021. https://doi. org/10.1109/mysurucon52639.2021.9641614. [5] a. pandey, n. gill, k. sai prasad nadendla, and i. s. thaseen, “identification of phishing attack in websites using random forest-svm hybrid model,” in international conference on intelligent systems design and applications, pp. 120–128, springer, 2018. https://doi.org/10.1007/ 978-3-030-16660-1_12. [6] a. ramana, k. l. rao, and r. s. rao, “stop-phish: an intelligent phishing detection method using feature selection ensemble,” social network analysis and mining, vol. 11, no. 1, pp. 1–9, 2021. https://doi.org/10.1007/s13278-021-00829-w. [7] h. abusaimeh and y. alshareef, “detecting the phishing website with the highest accuracy,” tem journal, vol. 10, pp. 947–953, 2021. https://www.ceeol.com/search/article-detail? id=955454. [8] n. tabassum, f. f. neha, m. s. hossain, and h. s. narman, “a hybrid machine learning based phishing website detection technique through dimensionality reduction,” in 2021 ieee international black sea conference on communications and networking (blackseacom), pp. 1–6, ieee, 2021. https://doi.org/10.1109/blackseacom52164.2021.9527806. [9] a. lakshmanarao, p. s. p. rao, and m. b. krishna, “phishing website detection using novel machine learning fusion approach,” in 2021 international conference on artificial intelligence and smart systems (icais), pp. 1164–1169, ieee, 2021. https://doi.org/10.1109/icais50930. 2021.9395810. paradigmplus (2022) 3:3 https://www.csoonline.com/article/2117843/what-is-phishing-examples-types-and-techniques.html https://www.csoonline.com/article/2117843/what-is-phishing-examples-types-and-techniques.html https://www.dataendure.com/wp-content/uploads/2021_cost_of_a_data_breach_-2.pdf https://www.dataendure.com/wp-content/uploads/2021_cost_of_a_data_breach_-2.pdf https://doi.org/10.1109/iccrd54409.2022.9730579 https://doi.org/10.1109/mysurucon52639.2021.9641614 https://doi.org/10.1109/mysurucon52639.2021.9641614 https://doi.org/10.1007/978-3-030-16660-1_12 https://doi.org/10.1007/978-3-030-16660-1_12 https://doi.org/10.1007/s13278-021-00829-w https://www.ceeol.com/search/article-detail?id=955454 https://www.ceeol.com/search/article-detail?id=955454 https://doi.org/10.1109/blackseacom52164.2021.9527806 https://doi.org/10.1109/icais50930.2021.9395810 https://doi.org/10.1109/icais50930.2021.9395810 hybrid approach for phishing website detection using classification algorithms 29 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [10] a. subasi and e. kremic, “comparison of adaboost with multiboosting for phishing website detection,” procedia computer science, vol. 168, pp. 272–278, 2020. https://doi.org/10.1016/ j.procs.2020.02.251. [11] a. zamir, h. u. khan, t. iqbal, n. yousaf, f. aslam, a. anjum, and m. hamdani, “phishing web site detection using diverse machine learning algorithms,” the electronic library, vol. 38, no. 1, pp. 65–80, 2020. https://doi.org/10.1108/el-05-2019-0118. [12] l. r. kalabarige, r. s. rao, a. abraham, and l. a. gabralla, “multilayer stacked ensemble learning model to detect phishing websites,” ieee access, vol. 10, pp. 79543–79552, 2022. https: //doi.org/10.1109/access.2022.3194672. [13] a. makkar, n. kumar, l. sama, s. mishra, and y. samdani, “an intelligent phishing detection scheme using machine learning,” in proceedings of the sixth international conference on mathematics and computing, pp. 151–165, springer, 2021. https://doi.org/10.1007/978-981-15-8061-1_ 13. [14] r. yang, k. zheng, b. wu, c. wu, and x. wang, “phishing website detection based on deep convolutional neural network and random forest ensemble learning,” sensors, vol. 21, no. 24, p. 8281, 2021. https://doi.org/10.3390/s21248281. [15] m. al-sarem, f. saeed, z. g. al-mekhlafi, b. a. mohammed, t. al-hadhrami, m. t. alshammari, a. alreshidi, and t. s. alshammari, “an optimized stacking ensemble model for phishing websites detection,” electronics, vol. 10, no. 11, p. 1285, 2021. https://doi.org/10.3390/ electronics10111285. [16] p. l. indrasiri, m. n. halgamuge, and a. mohammad, “robust ensemble machine learning model for filtering phishing urls: expandable random gradient stacked voting classifier (ergsvc),” ieee access, vol. 9, pp. 150142–150161, 2021. https://doi.org/10.1109/access.2021. 3124628. [17] m. alsaedi, f. a. ghaleb, f. saeed, j. ahmad, and m. alasli, “cyber threat intelligence-based malicious url detection model using ensemble learning,” sensors, vol. 22, no. 9, p. 3373, 2022. https://doi.org/10.3390/s22093373. [18] m. korkmaz, o. k. sahingoz, and b. diri, “detection of phishing websites by using machine learning-based url analysis,” in 2020 11th international conference on computing, communication and networking technologies (icccnt), pp. 1–7, ieee, 2020. https://doi.org/10.1109/ icccnt49239.2020.9225561. [19] a. b. shaik and s. srinivasan, “a brief survey on random forest ensembles in classification model,” in international conference on innovative computing and communications, pp. 253–260, springer, 2019. https://doi.org/10.1007/978-981-13-2354-6_27. [20] r. r. popat and j. chaudhary, “a survey on credit card fraud detection using machine learning,” in 2018 2nd international conference on trends in electronics and informatics (icoei), pp. 1120–1125, ieee, 2018. https://doi.org/10.1109/icoei.2018.8553963. [21] e. bujokas, “feature importance in decision trees.” https://towardsdatascience.com/ feature-importance-in-decision-trees-e9450120b445, 2022. paradigmplus (2022) 3:3 https://doi.org/10.1016/j.procs.2020.02.251 https://doi.org/10.1016/j.procs.2020.02.251 https://doi.org/10.1108/el-05-2019-0118 https://doi.org/10.1109/access.2022.3194672 https://doi.org/10.1109/access.2022.3194672 https://doi.org/10.1007/978-981-15-8061-1_13 https://doi.org/10.1007/978-981-15-8061-1_13 https://doi.org/10.3390/s21248281 https://doi.org/10.3390/electronics10111285 https://doi.org/10.3390/electronics10111285 https://doi.org/10.1109/access.2021.3124628 https://doi.org/10.1109/access.2021.3124628 https://doi.org/10.3390/s22093373 https://doi.org/10.1109/icccnt49239.2020.9225561 https://doi.org/10.1109/icccnt49239.2020.9225561 https://doi.org/10.1007/978-981-13-2354-6_27 https://doi.org/10.1109/icoei.2018.8553963 https://towardsdatascience.com/feature-importance-in-decision-trees-e9450120b445 https://towardsdatascience.com/feature-importance-in-decision-trees-e9450120b445 introduction related works dataset description methodology machine learning algorithms xgboost random forest (rf) support vector machine (svm) naive bayes (nb) logistic regression k-nearest neighbors (knn) decision tree (dt) adaboost hybrid model implementation results and discussion conclusion vol. 3, no. 2, pp 11 26, 2022 doi: 10.55969/paradigmplus.v3n2a2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . predictive analysis of mental health conditions using adaboost algorithm elizabeth oluyemisi ogunseye id 1, cecilia ajowho adenusi id 2, andrew c. nwanakwaugwu id 3,�, sunday adeola ajagbe id 4,�, and solomon o. akinola id 1 1university of ibadan, ibadan, nigeria elizabetholuyemisio@yahoo.com, solom202@yahoo.co.uk 2linux professional institute, ibadan, nigeria ceceresearch@gmail.com 3university of salford, greater manchester, united kingdom a.c.nwanakwaugwu@edu.salford.ac.uk 4ladoke akintola university of technology, ogbomoso, nigeria saajagbe@pgschool.lautech.edu.ng abstract the presented research responds to increased mental illness conditions worldwide and the need for efficient mental health care (mhc) through machine learning (ml) implementations. the datasets employed in this investigation belong to a kaggle repository named "mental health tech survey." the surveys for the years 2014 and 2016 were downloaded and aggregated. the prediction results for bagging, stacking, lr, knn, tree class, nn, rf, and adaboost yielded 75.93%, 75.93%, 79.89%, 90.42%, 80.69%, 89.95%, 81.22%, and 81.75% respectively. the adaboost ml model performed data cleaning and prediction on the datasets, reaching an accuracy of 81.75%, which is good enough for decision-making. the results were further used with other ml models such as random forest (rf), k-nearest neighbor (knn), bagging, and a few others, with reported accuracy ranging from 81.22% to 75.93% which is good enough for decision making. out of all the models used for predicting mental health treatment outcomes, adaboost has the highest accuracy. keywords: artificial intelligence ⋅ predictive analysis ⋅ mental health conditions ⋅ machine learning ⋅ adaboost algorithm received: 29 july 2022 ⋅ accepted: 25 august 2022 ⋅ published: 29 august 2022. 1 introduction mental health disorder (mhd) is also known as a mental illness disorder with diverse symptoms ranging from feeling down and sad, excessive worries or fears, sleeping issues, low energy, and significant tiredness. there are diverse kinds of mental health, which include bipolar disorder, anxiety disorder, suicidal ideation, and many more. according to rehm et al., [1], millions of people are suffering from mental disorders and if care is not taken early enough, it can lead to the progression of all kinds of disease and discomfort, which will then lead to severe treatments. and it is being treated and evaluated using subjective, self-reported, or questionnaire-based methods, which takes time and this article is distributed under the terms of the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. https://doi.org/10.55969/paradigmplus.v3n2a2 https://orcid.org/0000-0001-9086-9436 https://orcid.org/0000-0003-0522-8971 https://orcid.org/0000-0002-2929-9638 https://orcid.org/0000-0002-7010-5540 https://orcid.org/0000-0003-4034-8602 12 elizabeth oluyemisi ogunseye et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . can be difficult at times. mh issues have gotten to the systematic research stage. the world health organization (who) listed many mental conditions that cause depression, psychological problems, and schizophrenia [2]. the united nations stated that, according to their statistics, almost 20 million deaths are recorded every year from mental health disorders. in response to this, who stated that mh is now given an upper priority and therefore leads to more study and research on mh. universities and most citizens’ understanding and warning remain at the traditional level of psychological warning, which is limited and theoretical. in this new age of information technology (it), data mining, and big data, it has started to play a vital role in the mh field, most especially in the aspect of rising alarm of the early warning sigma or symptoms, and most institutions have realized the use of informa ionization of student management, which only entails the collection stage, storage means, student information management, and combining such with the literature [3, 4]. the vast increase in the usage of the internet has led to fast growth in data generation, which has led to the utilization of data mining technology for mining values from big data. such operations commence with classification, which apparently saves a lot of time and economic cost as in data analysis, tuning & fitting, predicting & forecasting processes, followed by regression operations. the methods that are most ideal for classification and regression operations are known as ensemble methods. this combines several models to form one good model via reducing variance and bias in order to enhance the prediction accuracy percentage [5]. common methods used for these are boosting [6] and bagging, of which boosting produces better results than bagging [7]. a boosting algorithm is generally a learning algorithm in the field of machine learning, which makes use of a "probably approximately correct" model designed by valiant. the algorithm was further improved by [8] and the improved version was named the adaboost algorithm. this algorithm has now been improved to a more vital feature classification algorithm in ml and is widely used for diverse areas like intrusion detection [9], feature extraction [10], image retrieval, object recognition [11], and other applications. the error rate increases during training as weights assigned to samples via adaboost rise and likewise fall as new weights fall [12]. ml tools have become useful tools for psychiatry in order to disentangle parameters associated with patient outcomes. the majority of ml research in psychiatry has focused on classification and diagnosis, and as a result, is prone to poor performance due to a lack of understanding of the application of data science and ml techniques, as well as corresponding diagnostic processes or psychiatric disorders. but advancement has been made in some areas via decision tree (dt) models in predicting suicide in u.s. military personnel and adolescents, and dt has been able to show how variables can be used to give insight into the models being used. this research looks into the ml algorithms in analyzing and predicting mh conditions and treatments via the use of the mental health tech survey dataset gotten from the kaggle repository. the remaining sections of this work are organized as follows. section 2 and 3 present related work and methodology respectively. section 4 contains the results and discussion, while sections 5 and 6 conclude the study and highlight future scopes. 2 related work in transforming mental healthcare, graham et al., [13] opined that ai technology holds amazing and promising roles in achieving solutions to many problems. the researchers reviewed 28 papers on ai and mental health that made use of brain imaging data, electronic health records (ers), novel smart monitoring systems, mood rating scales, and platforms on social media in classifying and predicting schizophrenia, health illness (depression), suicide attempts, and ideation. the outcome of the studies showed high accuracy and provided good examples of the potential of ai in healthcare for mental illness. according to madan et al., [14], there are vast real-world databases of health care records available worldwide that can be used to improve personalized medicine and predictive analysis. of these, the paradigmplus (2022) 3:2 predictive analysis of mental health conditions using adaboost algorithm 13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . major source of information collection in mental health has been text-based records. therefore, the main approach to detecting psychiatric disease is via text mining, according to the researchers. in this paper, 150 text files were manually used for psychiatric symptoms and attributes, and they were divided into two (training and test set). a system was designed to automatically detect the attributes, and the workflow made use of a neural network model. the overall result shows 86% of the f1 score with 91% of the independent test set. zhang et al., [15] were concerned about depression, which is one of the mental disorders that keeps increasing in its effect on a person’s physical health. the technology approach of ai has currently been implemented to aid mental health practitioners, most especially clinicians and psychiatrists, in decision-making via their patients’ medical records. the researchers explored children’s mental health and how to use ai to cope with mh issues in children. in order for it to be treated at an early stage, deep learning tools were proposed to be implemented in diagnosing and forecasting children’s mental health. a convolutional neural network (cnn) was first used to examine the deep parts of children’s behavior data features, followed by insights into the disruption, classification, and then finally, forecasting. the results showed that this current model had a 97.9% sensitivity ratio, 96.7% of the rate specified, 95.6% of ratio recall with 90.1% of precision ratio, and 95.6% of f rate measured with a 9.2% error rate, which is better than the existing system. one of the complex illnesses associated with varieties of clinical associations and socioeconomic status is mental illness and it’s being associated with diverse textual clinical data, coupled with those from social media and interviews [16]. natural language processing (nlp) is one of the methods used by researchers as one means to aid in early diagnosis and empower the medical healthcare system. overall, 399 papers from about 10,467 records were reviewed and the review results showed that there was an upward trend detected via nlp of mental illness. therefore, for further studies, a recommendation on the use of deep learning was made. sharma et al., [17] studied diagnosis of mental illnesses based on direct interviews designed with some set of questions, which mostly consume more time and human energy, because men’s health is beginning to gain more ground and the need for the use of machine learning (ml) is also beginning to arise, with the healthcare system not left behind. the researchers here then made use of ml in developing a model that can enhance the diagnosis of mental depression, and such a model was evaluated. a sample of about 11,081 records of dutch citizens was used and extreme gradient boosting (xgboost), which is one of the types of ml algorithm, was applied to each sample, and classification was carried out with an accurate result of 0.90. this research looks into mental health conditions by analyzing the survey of mental health tech to extract meaningful information, and this was achieved via the means of aggregation. the survey years 2014 and 2016 were merged together in order to extract more meaningful information, which they then used to predict the number of people responding to treatment. 3 methodology this section entails the method used in collating the information used for this research work and the flow of how the data collected will be utilized properly. the datasets used were downloaded from the kaggle repository and were titled mental health in tech survey for the years 2014 and 2016. the 2014 dataset has 1259 instances with 27 parameters. the 2016 dataset has 1433 instances with 62 parameters. both were used to carry out the analysis and prediction processes. the structure in figure 1 denotes the architecture of the mental health predictive analysis in this study hence, it shows how the whole system was implemented, commencing from when the dataset is able to be cleaned of such data in order to avoid missing values that may affect the accuracy of any models implemented, to the real analysis of all the parameters used in the survey, to the covariance matrix, correlation matrices, and probability of mental health occurrence, to how people respond to treatment, and predicting the respondents’ treatment with roc curve visualized results. paradigmplus (2022) 3:2 14 elizabeth oluyemisi ogunseye et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 1: architecture of mental health predictive analysis 3.1 algorithm of mhd with adaboost 1○ input: training dataset ={(m h1, t r1),(m h2, t r2), . . .(m hn, t rn)}; weak learn algorithm; ensemble size t. 2○ initialization: starting training set using distribution weight (see equation 1). w1c ∶ w 1 c = 1 n , c = 1, 2, . . ., n (1) 3○ do for e = 1, 2, . . . , t – compute a weak classifier based on the classifier algorithm with present weight we – compute weighted error during training (see equation 2). ϵe ∶ ϵe = n ∑ c=1 wec ; trc≠he (mh c) (2) – update weight of training instances (see equation 3). we+1c = w e c . exp(−∝e.trc.he(mh c) m h ze , whereze = n ∑ c=1 .wec . exp(−∝e.trc.he (mh c)) (3) paradigmplus (2022) 3:2 predictive analysis of mental health conditions using adaboost algorithm 15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4○ output: ensemble classifier (see equation 4). f (mh)= sign ( t ∑ e=1 .∝e.ht (mh)) (4) 3.2 models used for mental health conditions 3.2.1 adaboost adaptive boosting is a classification algorithm that forms a committee of weak classifiers and enhances ml [18] performance algorithms by combining them together into a strong classifier. the main goal of this classifier is to enhance the weight of uncategorized points and decrease the weight of categorized points. some of the advantages of adaboost are its ease of use and the accuracy of the weak classifier, and some of the disadvantages are the need for quality data and optimization of hyperparameters. another name for the tree generated is called decision stump [19], and the model will keep training till a lower error is achieved [20]. 3.2.2 random forest (rf) classifier this classifier made use of training data to model many tree structures, and each tree suggested a class as an output, with vast numbers of results selected as the overall outcome [21]. the building structure requires specific numbers of trees and, as such can be used as a means of bagging and aggregating bootstrap data. this can be used to reduce variance in the final result. 3.2.3 neural network (nn) an artificial neural network (ann) refers to a biologically inspired sub-field of artificial intelligence modeled after the brain. artificial neural networks (anns) are computer networks that are based on biological principles. multilayer perceptrons (mlps) with back propagation learning algorithms are the focus of this chapter among the numerous forms of anns [22]. mlps are supervised anns with three layers: input, hidden, and output. they are the most commonly used anns for a wide range of tasks. we go over the structure, method, data pretreatment, overfitting, and sensitivity analysis of mlps, among other things. advantages: the ability to work with insufficient knowledge and good fault tolerance. disadvantages: the difficulty of showing the problem to the network and hardware dependence [23]. 3.2.4 decision tree (dt) this is a supporting tool that has a tree-shaped structure and models the cost of resources, possible outcomes, consequences, and utilities. it also provides a means of presenting the algorithm via a control statement. dt involves the use of a flowchart, which has nodes that denote each stage, and each branch denotes the attribute result, while the rule of classification is the path from the leaf to root. the tree structure denotes classification and if-then rules, which are quite easy to understand. the computation methods involved are quite short and efficient. it can be used in computing symptoms related to mental health. mining can be achieved with the use of knowledge and law via algorithms and the provision of student mental health for efficient decision-making for student education [24]. 3.2.5 k-nearest neighbor (knn) one of the simpler machine learning methods is this algorithm. many in the industry favor it because of how simple it is to use and how quickly calculations can be completed. using the points that are paradigmplus (2022) 3:2 16 elizabeth oluyemisi ogunseye et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . most similar to it, the knn model categorizes data points. it "informed guesses" what an unclassified point should be classed as using test data. simple recommendation systems, image recognition software, and decision-making models frequently use knn [25, 26]. 4 result and discussion this section analyzed, visualized, and predicted mental health using ml models, and the adaboost model ultimately performed better than other ml models in this study. the result was further validated with other models and the ada result had an accuracy of 81.75% while others were lower than that. figure 2 is the correlation matrix of mh parameters and figure 3 is the respondents by countries with tech flag and company size orange. most respondents in the survey are in tech. most of them are from the us and work in medium or large companies. even though the description of the research appeared to be global, which would have been a great way to amplify the problem in various communities, the research ended up focusing on the usa and a few european countries. as expected, most of the respondents are male, this happened because the tech industry is occupied by a big majority of men. the proportion of females figure 2: correlation matrix of mh parameters paradigmplus (2022) 3:2 predictive analysis of mental health conditions using adaboost algorithm 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 3: respondents by countries with tech flag and company size orange figure 4: gender by countries with overall proportional (only techs) figure 5: mental health disorder in tech companies with overall percentage proportion (currently) in tech in this survey in the united states (us) is the biggest one out of all countries. figure 4 shows the gender of the participants per country, and figure 5 is the mental health disorder in tech. companies with overall percentage proportion. the results in figure 6 show that there are tech workers with paradigmplus (2022) 3:2 18 elizabeth oluyemisi ogunseye et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 6: mental health disorder in tech with past overall percentage proportion mhd more in the us followed by (the united kingdom), from the experiment, the result from the tech dataset also reveals that the uk has more mhd than any other countries that participated in the survey with over 41.8% acknowledged having mhd. the response was either ’yes’, ’no’, or ’maybe’. there were tech workers with mhd more in the us followed by (the united kingdom) uk in the past than any other countries that participated in the survey with over 53.7% acknowledged having mhd. so, the rate dropped from 53.7% to 41.8% from the present analysis. figure 7 shows the result of the extraction of important features from the tech dataset. these include age, gender, family history, benefits, care options, anonymity, leave, and work interference. this highlighted the importance of features extracted for further analysis of the mental health disordered, figure 7: importance features extraction paradigmplus (2022) 3:2 predictive analysis of mental health conditions using adaboost algorithm 19 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 8: mh considerations by companies which ranges from age down to work interface to arrive at a reasonable conclusion. the result in figure 8 shows the results of mh considerations by companies. that about 55.7% companies have mh coverage provided for their employees, 21.7% provided resources, 21.4% provided anonymity, 35.2% figure 9: consequences of mhd on career observed paradigmplus (2022) 3:2 20 elizabeth oluyemisi ogunseye et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 10: being open about mental health with friends and family figure 11: treatment correlation matrix parameters requested for medical leave, with 24.3 having both mental and physical health of equal importance and only 6% had negative consequences of mhd over their work and 94% were able to manage it effectively. in the main time, there are ripple effects of these considerations on the overall output of these companies. figure 9 shows that only 38.8% think mhd may hurt their careers while 42.2% were not so sure. 2.3% has not actually experienced mhd, 10.1% think no, 6.6% yes it has. therefore, mhd will continue to affect the career of work since consciousness is not there to address it. when considering openness about mh with friends and family, figure 10 shows that only 18.3% were very open to their friends and family, 45.1% are sometimes open 9.1% neutral, 14.8% sometimes not open, while 5.1% were not opened at all and 7.6% na to me. therefore, mhd will continue to affect the career of workers since open to addressing the issue. figure 11 shows the relationship between the mhd parameters used for the analysis and prediction while using ml models figure 12 visualized the histogram chart showing those with mhd and are undergoing treatment and those that have not appeared, those that are on treatment are more than those that have not. paradigmplus (2022) 3:2 predictive analysis of mental health conditions using adaboost algorithm 21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 12: separate by treatment or not figure 13: probability of age-range of mental health condition with care option based on gender figure 13 results shows 100% (1.0) probability of female between the age-range of 66-100 having mhd 31-65 age-range shows about 0.72 probability of female having mhd, 21-30 age range with 0.7 probability of mhd and 07. probability of females having mhd of 0-20 age-range. the actual result of mhd with treatment is presented as follows, while figure 14 shows adaboost predicted result of mhd with treatment. ########### adaboosting ############### accuracy: 0.81746031746 null accuracy: 0 191 1 187 name: treatment, dtype: int64 percentage of ones: 0.4947089947089947 percentage of zeros: 0.5052910052910053 true: [0 0 0 0 0 0 0 0 1 1 0 1 1 0 1 1 0 1 0 0 0 1 1 0 0] pred: [1 0 0 0 0 1 0 1 1 1 0 1 1 0 1 1 1 1 0 0 0 0 1 0 0] paradigmplus (2022) 3:2 22 elizabeth oluyemisi ogunseye et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 14: adaboost predicted result the performance of eight ml models used in this study is presented in table 1, and it shows how adaboosting outperformed other ml models in this study. table 1: table style s/n ml models accuracy (%) 1 bagging 75.93 2 stacking 75.93 3 lr 79.89 4 knn 80.42 5 tree class 80.69 6 nn 80.95 7 rf 81.22 8 adaboost 81.75 5 conclusion the study presents an ml implementation that impacts how mental disorders are being diagnosed. to this end, the kaggle repository has been used as a training/testing dataset. we attempted to classify the patients with different supervised algorithms and concluded that adaboost was the most successful tool for the task. in other words, this research work has been able to explore the mental health (mh) tech survey dataset, and diverse insights have been extracted and visualized from the dataset. it shows the application of ml will aid in the faster collection, sorting, and understanding of the data, as well as prediction. this visualization includes the gender, age, where the survey took place, the reaction of people and workers to their employees having mh issues, how their work has been affected, treatment or not issues down to prediction using ml models. the accuracy of prediction results was compared among the eight ml models used, the results of the prediction for bagging, stacking, lr, knn, tree class, nn, rf, and adaboost yielded 75.93%, 75.93%, 79.89%, 90.42%, 80.69%, 89.95%, 81.22%, and 81.75% respectively. the results also show that more males responded than females and from all the countries that the survey was done, respondents from the united states responded well. work and family history were also reflected in the analysis, with the majority of the respondents having a history in their family, and most of them were under one care or the other, esparadigmplus (2022) 3:2 predictive analysis of mental health conditions using adaboost algorithm 23 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . pecially the female respondents. the results further show that mh often interfaces with their work performance. and the predictions made with ml show the probability of respondents responding to treatments keeps increasing. 6 future work kaggle is a bonafide source of data. however, the data set seems very limited when it comes to mh dataset as it contains mh datasets of a few of the developed countries. hence, there is a need to populate mh dataset for an informed decision-making process. this research can also be further improved upon by making use of the mental health tech survey 2020 dataset if one exists or conducting another survey for 2020 and then applying another model using an ensemble model with natural language processing (nlp) and all the missing values can be further removed for more accurate results. the survey in 2014 and 2016 was great. replica of such survey is suggested, now that during the covid-19 restrictions and social distancing is prevailing and compare the results authors’ information – elizabeth oluyemisi ogunseye obtained b.sc in computer science from ambrose alli university, ekpoma in 2007, a master’s degree in computer systems in 2009, another master’s degree in computer science in 2016, and currently, she is doing a ph.d. at the department of computer science at the university of ibadan, nigeria. her research interests include health informatics, artificial intelligence, data science, and data mining – cecilia ajowho adenusi is a postgraduate student of the department of computer science at the federal university of agriculture, abeokuta, ogun state, nigeria. she currently works at linux professional institute, nigeria master affiliate as the head of training at the institute. she is a member of the computer professional council of nigeria(cpn), member of nigeria computer society (ncs), member of academia in information technology (aitp), and member of nigerian women in information technology(niwiit). – andrew c. nwanakwaugwu is a scholarly data science enthusiast with years of it (information technology) related working experience. he is currently working as a pt operations advisor at hmrc united kingdom (uk). he worked as a data scientist at greater manchester ai foundry where he used artificial neural networks (ann) in deep learning and the development of a customized speech synthesis system with an interactive user interface. he also worked at jamasoft concept as a system analyst/data scientist, media/musical instrumentalist at rccg manchester. he studied computer engineering in his first degree and data science in his second degree in nigeria and the university of salford, uk respectively. – sunday adeola ajagbe is a ph.d. candidate in computer engineering at the ladoke akintola university of technology (lautech), ogbomoso, nigeria. he obtained an m.sc. and b.sc. in information technology and communication technology respectively at the national open university of nigeria (noun), and his postgraduate diploma in electronics and electrical engineering at lautech, and higher national diploma in electrical and electronics engineering, the polytechnic, ibadan. his specialization includes artificial intelligence (ai), natural language processing (nlp), information security, quantum computing, data science, internet of things (iot), and smart solution. he is also licensed by the council regulating engineering in nigeria (coren) as a professional electrical engineer, student member of the institute of electrical and electronics engineers (ieee), international association of engineers (iaeng), and corporate member of the nigeria computer society (ncs). paradigmplus (2022) 3:2 24 elizabeth oluyemisi ogunseye et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . – solomon o. akinola is a professor of computer science in the department of computer science at nigeria’s premier university, the university of ibadan, nigeria. he has a ph.d. in computer science at the university of ibadan, nigeria. he researches in the areas of software engineering and data mining. authors’ contributions – elizabeth oluyemisi ogunseye participated in conceptualization, review, validation, and supervision, – cecilia ajowho adenusi participated in conceptualization, writhing of the original draft, methodology, and software development. – andrew c. nwanakwaugwu participated in software design, project administration, and software development. – sunday adeola ajagbe participated in the methodology and writing of the original draft. – solomon o. akinola participated in editing, review, and supervision. competing interests the authors declare that they have no competing interests. funding no funding was received for this project. references [1] j. rehm and k. d. shield, “global burden of disease and the impact of mental and addictive disorders,” current psychiatry reports, vol. 21, no. 2, pp. 1–7, 2019. [2] g. singer and m. golan, “applying data mining algorithms to encourage mental health disclosure in the workplace,” international journal of business information systems, vol. 36, no. 4, pp. 553– 571, 2021. [3] m. fokkema, j. edbrooke-childs, and m. wolpert, “generalized linear mixed-model (glmm) trees: a flexible decision-tree method for multilevel and longitudinal data,” psychotherapy research, vol. 31, no. 3, pp. 329–341, 2021. [4] s. a. ajagbe, i. r. idowu, j. oladosu, and a. adesina, “accuracy of machine learning models for mortality rate prediction in a crime dataset,” international journal of information processing and communication (ijipc), vol. 10, no. 1, pp. 150–160, 2020. [5] m. zounemat-kermani, o. batelaan, m. fadaee, and r. hinkelmann, “ensemble machine learning paradigms in hydrology: a review,” journal of hydrology, vol. 598, p. 126266, 2021. [6] p. karimi and h. jazayeri-rad, “comparing the fault diagnosis performances of single neural networks and two ensemble neural networks based on the boosting methods,” journal of automation and control, vol. 2, no. 1, pp. 21–32, 2014. [7] d. opitz and r. maclin, “popular ensemble methods: an empirical study,” journal of artificial intelligence research, vol. 11, pp. 169–198, 1999. paradigmplus (2022) 3:2 predictive analysis of mental health conditions using adaboost algorithm 25 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [8] m. payal, a. kumar, and s. a. ajagbe, “support vector machines (svms) based advanced healthcare system using machine learning techniques,” international journal of innovative research in computer and communication engineering, vol. 10, no. 5, pp. 3007–3014, 2022. [9] y. zhou, t. a. mazzuchi, and s. sarkani, “m-adaboost-a based ensemble system for network intrusion detection,” expert systems with applications, vol. 162, p. 113864, 2020. [10] b. liu, c. liu, y. xiao, l. liu, w. li, and x. chen, “adaboost-based transfer learning method for positive and unlabelled learning problem,” knowledge-based systems, vol. 241, p. 108162, 2022. [11] g. hu, c. yin, m. wan, y. zhang, and y. fang, “recognition of diseased pinus trees in uav images using deep learning and adaboost classifier,” biosystems engineering, vol. 194, pp. 138–151, 2020. [12] n. gupta and s. nigam, “crude oil price prediction using artificial neural network,” procedia computer science, vol. 170, pp. 642–647, 2020. [13] s. graham, c. depp, e. e. lee, c. nebeker, x. tu, h.-c. kim, and d. v. jeste, “artificial intelligence for mental health and mental illnesses: an overview,” current psychiatry reports, vol. 21, no. 11, pp. 1–18, 2019. [14] s. madan, f. j. zimmer, h. balabin, s. schaaf, h. fröhlich, j. fluck, i. neuner, k. mathiak, m. hofmann-apitius, and p. sarkheil, “deep learning-based detection of psychiatric attributes from german mental health records,” international journal of medical informatics, vol. 161, p. 104724, 2022. [15] x. zhang, r. wang, a. sharma, and g. g. deverajan, “artificial intelligence in cognitive psychology—influence of literature based on artificial intelligence on children’s mental disorders,” aggression and violent behavior, p. 101590, 2021. [16] t. zhang, a. m. schoene, s. ji, and s. ananiadou, “natural language processing applied to mental illness detection: a narrative review,” npj digital medicine, vol. 5, no. 1, pp. 1–13, 2022. [17] a. sharma and w. j. verbeke, “improving diagnosis of depression with xgboost machine learning model and a large biomarkers dutch dataset (n= 11,081),” frontiers in big data, p. 15, 2020. [18] c. a. adenusi, o. r. vincent, j. bolarinwa, and o. oluwakemi, “predicting the pandemic effect of covid 19 on the nigeria economic, crude oil as a measure parameter using machine learning,” in decision sciences for covid-19, pp. 79–93, springer, 2022. [19] a. s. hussein, t. li, c. w. yohannese, and k. bashir, “a-smote: a new preprocessing approach for highly imbalanced datasets by improving smote,” international journal of computational intelligence systems, vol. 12, no. 2, pp. 1412–1422, 2019. [20] p. c. reddy, r. m. s. chandra, p. vadiraj, m. a. reddy, t. mahesh, and g. s. madhuri, “detection of plant leaf-based diseases using machine learning approach,” in 2021 ieee international conference on computation system and information technology for sustainable solutions (csitss), pp. 1–4, ieee, 2021. [21] m. sarveshvar, a. gogoi, a. k. chaubey, s. rohit, and t. mahesh, “performance of different machine learning techniques for the prediction of heart diseases,” in 2021 international conference on forensics, analytics, big data, security (fabs), vol. 1, pp. 1–4, ieee, 2021. [22] o. i. abiodun, a. jantan, a. e. omolara, k. v. dada, n. a. mohamed, and h. arshad, “stateof-the-art in artificial neural network applications: a survey,” heliyon, vol. 4, no. 11, p. e00938, 2018. paradigmplus (2022) 3:2 26 elizabeth oluyemisi ogunseye et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [23] j. zhang, z. gu, j. jang, h. wu, m. p. stoecklin, h. huang, and i. molloy, “protecting intellectual property of deep neural networks with watermarking,” in proceedings of the 2018 on asia conference on computer and communications security, pp. 159–172, 2018. [24] m. sheikhhassan, m. sadeghzadeh, and m. faghihi, “analysis of health tourist’s behaviors using data mining method (international tourists),” razavi international journal of medicine, vol. 9, no. 2, pp. 48–55, 2021. [25] d. prasad, s. k. goyal, a. sharma, a. bindal, and v. s. kushwah, “system model for prediction analytics using k-nearest neighbors algorithm,” journal of computational and theoretical nanoscience, vol. 16, no. 10, pp. 4425–4430, 2019. [26] y. wang and z.-o. wang, “a fast knn algorithm for text categorization,” in 2007 international conference on machine learning and cybernetics, vol. 6, pp. 3436–3441, ieee, 2007. paradigmplus (2022) 3:2 introduction related work methodology algorithm of mhd with adaboost models used for mental health conditions adaboost random forest (rf) classifier neural network (nn) decision tree (dt) k-nearest neighbor (knn) result and discussion conclusion future work vol. 2, no. 3, pp 1 13, 2021 doi: 10.55969/paradigmplus.v2n3a1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . improving the robustness of graphsaint via stability training yuying wang id 1,�, huixuan chi id 1, and qinfen hao id 1 1institute of computing technology, chinese academy of sciences, beijing, china wangyuying@ict.ac.cn, chihuixuan21@mails.ucas.ac.cn, haoqinfen@ict.ac.cn abstract graph neural networks (gnns) field has a dramatic development nowadays due to the strong representation capabilities for data in non-euclidean space, such as graph data. however, as the scale of the dataset continues to expand, sampling is commonly introduced to obtain scalable gnns, which leads to the instability problem during training. for example, when graph sampling based inductive learning method (graphsaint) is applied for the link prediction task, it may not converge in training with a probability range from 0.1 to 0.4. this paper proposes the improved graphsaints by introducing two normalization techniques and one graph neural network (gnn) trick into the traditional graphsaint to solve the problem of the training stability and obtain more robust training results. the improved graphsaints successfully eliminate the instability during training and improve the robustness of the traditional model. besides, we can also accelerate the training procedure convergence of the traditional graphsaint and obtain a generally higher performance in the prediction accuracy by applying the improved graphsaints. we validate our improved methods by using the experiments on the citation dataset of open graph benchmark (ogb). keywords: graph neural networks (gnns) ⋅ training stability ⋅ normalization techniques ⋅ gnn tricks ⋅ link prediction received: 31 october 2021 ⋅ accepted: 19 november 2021 ⋅ published: 2 december 2021. 1 introduction nowadays, the graph neural networks (gnns) proposed in 2009 [1] has been widely used for processing the data in the non-euclidean space, especially the graph data. because of the powerful processing capabilities for the unstructured data, gnns are widely used in many areas such as social networks [2, 3], drug discovery [4], and recommendation [5, 6]. the existing gnn models can be roughly classified into four categories, one of which is convolutional graph neural networks (convgnns) [7]. convgnns can be further classified into two categories, spectral-based convgnns and spatial-based convgnns. graph convolution network (gcn) is a typical spectral-based convgnn model [8, 9]. to efficiently train gcn and improve the generalization ability of gcn for new nodes, the graph sampling based inductive learning method (graphsaint) [10] is proposed. graphsaint uses a sampling method called the subgraph sampling method to generate a serial of subgraphs from the whole graph and then build a gcn on the subgraph. this subgraph sampling method alleviates the this article is distributed under the terms of the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. https://doi.org/10.55969/paradigmplus.v2n3a1 https://orcid.org/0000-0003-3097-6808 https://orcid.org/0000-0002-5825-3452 https://orcid.org/0000-0002-1545-1600 2 yuying wang et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . problem of the neighbor explosion so that the number of neighboring nodes no longer increases exponentially with the number of layers in graphsaint. moreover, graphsaint has a stronger processing capability for large graphs by sampling multiple subgraphs from the original dataset to construct minibatch for training compared with the graph sample and aggregate (graphsage) algorithm [11]. in general, nodes with a broader influence on each other can be selected to form a subgraph with a higher probability. however, the subgraph sampling method applied in graphsaint leads to a different node sampling probability and introduces bias into the mini-batch estimator, which makes the training more difficult. to deal with this problem, the normalization techniques are proposed and applied in [10] to make sure that the feature learning will not give priority to the more frequently sampled nodes. therefore, graphsaint can achieve a better accuracy score on the classification task. however, experiments show that the stability problem appears during training when graphsaint is applied to deal with the link prediction task [12, 13] on the citation dataset of the standard open graph benchmark (ogb) [14]. unlike the node classification task, the main task of link prediction is to predict whether there is a link between two nodes in a network. this stability problem during training means that the normalization techniques in [10] are insufficient to guarantee the training quality for link prediction. so the training loss may fall into non-convergence, which indicates that the value of the training loss will suddenly rise and remain unchanged. training stability is important to the development of gnns. we can get some inspiration in training stability from the cnn training methods. stochastic gradient descent and its variants such as momentum [15] and adagrad [16] have been widely used to train the neural networks. as the number of neural network layers gradually increases, changes to the neural network parameters will amplify [17]. when constantly adapting to the new distribution, the distributions of layers’ inputs present a problem called covariate shift [18], which is bad for neural network convergence. ioffe and szegedy presented the batch normalization (bn) to reduce the internal covariate shift and accelerate the convergency of the deep neural nets [17]. this technique makes use of the mean and variance to normalize the data values over each mini-batch, which brings a higher learning rate and drops the dropout [19]. besides bn, layer normalization (ln) is also widely used among normalization techniques [20], but it uses a different normalization method from bn. however, the effective normalization techniques have drawn little attention in gnns due to the fewer network layers [7]. as the graph becomes larger and larger, the gnn models are more complicated, which leads to the stability problem during training. thus, the application of the normalization techniques is of great significance to the robustness in training on large graphs. besides, the tricks of gnns have drawn more attention in recent years. the presence of the tricks helps improve the accuracy of the gnn algorithms significantly on the ogb datasets [21, 22]. however, the role of gnn tricks on model stability needs to be further verified. therefore, to solve the stability problem in the training process of the traditional graphsaint for the link prediction task, we propose the improved graphsaints by adjusting two normalization techniques, that is bn and ln, and one gnn trick to the training and inference process of the traditional graphsaint based on the citation dataset of ogb in this paper. by applying the three techniques to the model, we eliminate the instability of the traditional graphsaint during training successfully. moreover, we also realize a reduction in the training time and improve the accuracy under the premise of maintaining the original link prediction accuracy with the help of the improved graphsaints. we validate the effectiveness of our methods by the citation dataset of ogb. inspired by [23], the paper organizes as follows: firstly, section 2 is the related work about the traditional graphsaint, especially its sampling strategy. besides, we briefly present two normalization techniques, which is bn and ln, and a gnn trick; next, section 3 describes our improved graphsaints and the processes of training; then, section 4 shows the related experiment results based on the citation dataset of ogb; at last, conclusions are given in section 5. paradigmplus (2021) 2:3 improving the robustness of graphsaint via stability training 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 related work 2.1 the sampling strategy of the traditional graphsaint since gcn aggregates one-hop neighbors’ information via the adjacent matrix [8, 9], it has a low generalization ability as mentioned in section 1, that is, gcn can not deal with the problem of adding a new node to a graph. if we want to obtain the node representation of the new node, we need to update the adjacent matrix and re-train the gcn model to update the model parameters. therefore, graphsage is proposed to improve the generalization ability of gcn [11]. graphsage makes use of the neighbor sampling strategy and defines a new learnable aggregation function to aggregate neighbor nodes’ information. the process of generating a target node embedding avoids using the adjacent matrix and the time of retraining the model. instead of the neighbor sampling strategy, graphsaint achieves the sampling of large images by proposing a sampler called sample [10]. this sampling method is called the subgraph sampling method, which can help graphsaint deal with the neighbor explosion problem. besides, similar to graphsage, graphsaint also has the generalization ability for new nodes. the sampler can preserve the connectivity characteristic of the graph. therefore, bias in the minibatch estimation will be almost inevitably introduced by the sampler. therefore, the determination of sampling probability is the critical step to realizing the subgraph sampling method. in [10], the sampling probabilities of each node, edge, and subgraph are all well-estimated by deriving the corresponding formulas. furthermore, the subgraphs obtained by sampling will be used for the graphsaint training. 2.2 the typical normalization techniques due to the change of the distributions during training, the internal covariate shift appears in a deep network. so the normalization techniques are used to eliminate this shift and achieve a faster training time [24, 25]. in the following, we mainly introduce two normalization techniques, which is bn and ln. bn presented in [17] is a typical normalization technique for mini-batch. bn follows the mathematical statistics: firstly, the mean is calculated over a mini-batch; next, the variance is calculated by the mean; then, each data over a mini-batch can be normalized by subtracting the mean and then divided by the variance; finally, the scale and shift parameters are learned for each data point over a mini-batch. in bn, the mean and the variance are calculated for different neuron unit inputs, and the inputs in the same batch have the same mean and variance. therefore, bn is sensitive to the size of the batch size. since the mean and the variance are calculated on one batch, if the batch size is too small, the calculated mean and variance are not enough to represent the entire data distribution. different from bn, ln calculates the mean and variance of the input of all neuron units in a specific layer of the deep network to perform the normalization operation and does not depend on the batch size. therefore, the input of the neuron units in the same layer in ln has the same mean and variance, and different input samples have unequal means and variances. 2.3 free large-scale adversarial augmentation on graphs (flag) as a gnn trick, flag is a free large-scale adversarial augmentation on graphs to alleviate overfitting. different from the existing literature of data augmentation by changing the graph structures, flag is conducted in the input node feature space. standard adversarial training aims to deal with an optimization problem [26]. this kind of optimization problem can be reliably solved by making use of both stochastic gradient descent (sgd) and projected gradient descent (pgd), which may cause the high computational cost and the loss of the model accuracy [27]. paradigmplus (2021) 2:3 4 yuying wang et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . flag is an adversarial data augmentation on graphs. it can improve the model accuracy and effectively enhance the generalization performance of the model. flag mainly achieves data augmentation by adding the small gradient-based adversarial perturbations to the input node features and preserving the graph structures. flag is a valuable exploration of the power of adversarial data augmentation on graphs. the general algorithm flow of flag is as follows: assume that each epoch contains m steps. in each epoch, we first define a perturbation matrix named pert. pert that has a gradient is the same shape as the input x and obeys a uniform distribution. then pert as perturbations and x are both sent to the model for training. the training has a total of m steps. we manually update pert according to the gradient of pert and then clear the gradient of pert in each step. finally, after finishing the m steps, we accumulate the m gradient values, and then the model parameters are all updated by the backpropagation. 3 methodology as mentioned in section 1, the original normalization techniques of the traditional graphsaint, which are valid for the node classification task, are insufficient to improve the training quality for the link prediction task on ogb. the traditional graphsaint needs to be improved to obtain more stable training results. therefore, we proposed three strategies, including bn, ln, and flag, to improve the traditional graphsaint and achieve more stable training results. according to the basic principle, the three strategies can be divided into two types: one is the normalization techniques including bn and ln, the other is the gnn trick named flag. based on the connectivity rules of the nodes, the sampled subgraphs in graphsaint can get the edge sampling probability with the minimum variance. in contrast, for node selection, the sampled subgraphs use the random node sampler. so the node feature data of each sampled subgraph do not obey the standard normal distribution. therefore, the normalization techniques bn and ln are used to normalize the mean and variance of the input data from different perspectives. besides, as a data augmentation method, the gnn trick named flag can enhance the model’s anti-interference ability against the input data by introducing small gradient-based adversarial perturbations into the input node feature data. in the following, we will introduce the two types of improved graphsaints in detail. 3.1 the improved graphsaints with the normalization techniques in this section, we will firstly introduce the first improved graphsaint with bn. this improved graphsaint is firstly introduced in [28]. here, we have made some necessary supplementary explanations based on the original version. the improved graphsaint with bn realizes that the inputs in the same batch have the same mean and variance. assume the graph dataset to be processed is a whole graph ζ = (v, ξ) with n nodes v ∈ v , edges (vi, vj) ∈ ξ, for the node vi in a sampled subgraph ζs of ζ according to sample, its feature hi,s has d elements. to normalize the input data distribution, the input node feature vector can be normalized by ĥi,s = hi,s − µbs√ (σbs) 2 (1) µbs = 1 d d ∑ i=1 hi,s σ b s = ¿ ááà1 d d ∑ i=1 (hi,s − µbs) 2 (2) where µbs and σbs are the mean and the variance for the node vi in the node feature dimension and computed over the training data set. therefore, after bn, the inputs in the same batch have the same mean and variance. paradigmplus (2021) 2:3 improving the robustness of graphsaint via stability training 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . however, the improved graphsaint with ln does not perform the same normalized operation in a batch different from the improved graphsaint with bn. this improved method with ln fixes the mean and the variance of the summed inputs in a specific layer of the deep network to reduce the covariate shift. thus, the ln can be calculated as follows: ĥi,s = hi,s − µls√ (σls) 2 (3) µls = 1 h h ∑ i=1 hi,s σ l s = ¿ ááà 1 h h ∑ i=1 (hi,s − µls) 2 (4) where µls and σls are the mean and the variance over all the hidden units in the same layer, h is the number of hidden units in a layer. therefore, the input of neuron units in the same layer in ln has the same mean and variance. because the different inputs have different means and variances, the input data in the same batch have unequal means and variances. the whole training process of the improved graphsaint with one of the two normalization techniques (bn or ln) is illustrated in algorithm 1. we can see that when using the two normalization techniques to improve graphsaint, a basic flow of the algorithm is roughly the same, and the only thing you need to do is to choose one of the two normalization techniques where the normalization technique is applied. the specific process of algorithm 1 is as follows: algorithm 1 the improved graphsaint during training with the normalization technique (bn or ln) on the citation dataset of ogb input: training graph ζ = (v, ξ); graphsaint sampler output: loss and accuracy; gcn model with trained weights pre-processing: directed graph to undirected graph; sampled subgraph ζs of ζ for each subgraph ζs do gcn construction on ζs and before each active layer relu do apply the normalization technology (bn or ln) on the output of the convolutional layer forward propagation to calculate loss according to mrr backward propagation to update weights end for at the beginning of the training, a directed graph is converted to an undirected graph in the stage of the pre-processing on ζ. and sample is conducted to obtain the sampled subgraph ζs [10]. next, an iterative training process is performed via sgd to obtain the updated weights. for each iteration, we use an independent subgraph ζs. in the iterative loop, the normalization technique (bn or ln) is applied on the output of the convolutional layer of gcn, which is also the input of the relu layer. then the modified gcn on ζs is conducted to rank the missing references and mean reciprocal rank (mrr) is used to calculate the loss in the process of the forward propagation. in mrr, the score of the nth matched result is 1/n and the final score is the sum of all scores. finally, the backward propagation is carried out to update weights. besides, although the subgraphs are used for training as mentioned in section 2.1, the whole graph data are used to obtain the reference result. therefore, during inference, the normalization operations should be conducted independently. 3.2 the improved graphsaint with a gnn trick as mentioned in section 2.3, the gnn trick named flag is a graph data augmentation method that helps relieve overfitting in the training process of a gnn model. flag enhances the model robustness by introducing small gradient-based adversarial perturbations into the input node features. the paradigmplus (2021) 2:3 6 yuying wang et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . algorithm 2 the improved graphsaint during training with a gnn trick named flag on the citation dataset in ogb input: training graph ζ = (v, ξ); graphsaint sampler output: loss and accuracy; gcn model with trained weights pre-processing: directed graph to undirected graph; sampled subgraph ζs of ζ for each subgraph ζs do introduce the small gradient-based adversarial perturbations into the input node features gcn construction on ζs forward propagation to calculate the loss under the premise of the perturbations according to mrr backward propagation to update weights end for improved graphsaint with the gnn trick named flag can solve the stability problem in the training process. besides, it also can improve the accuracy of the traditional graphsaint. the whole training process of the improved graphsaint with a gnn trick named flag is illustrated in algorithm 2. we can see that algorithm 2 has the same pre-processing as algorithm 1. then an iterative training process is also conducted via sgd to update model weights, and each iteration uses an independent subgraph ζs. next, in contrast, algorithm 2 introduces small gradient-based adversarial perturbations into the input node features. then the input node features with perturbations are fed into the gcn model as input, and gcn construction is performed. finally, under the premise of the perturbations, the loss can also be calculated according to mrr. 4 experiments in this section, to verify the effectiveness of the improved graphsaints, we choose the link prediction task based on the citation dataset of ogb (ogbl-citation). before the emergence of ogb, the main characteristic of the existing graph datasets is small. the small graph datasets are far from the web-scale graphs in the real world, which is harmful to the data-driven model research and practical problem solving, and the model trained with such a small dataset is also called a toy model. such toy models may have similar statistical results on these small datasets, which makes the models hard to distinguish. besides, such toy models are also hard to extend to large graphs. there is no doubt that ogb makes toy models a thing of the past. besides, ogb also gives a unified experimental standard, including the data set division method and the evaluation and verification standard. the datasets contained in ogb can be divided into three groups. each group corresponds to a specific graph machine learning task. there are three different tasks, which are node classification, link prediction, and graph classification. because the citation dataset that we choose is used for the link prediction task, citation can also be expanded as ogbl-citation by adding a prefix. the ogblcitation dataset is a directed graph that represents the citation network between subsets of papers extracted from microsoft academic graph (mag) [29]. each node is a paper with 128-dimensional word2vec features that summarize its title and abstract. each directed edge indicates that one paper cites another paper. all nodes are also accompanied by meta-information indicating the year of publication of the corresponding paper. the purpose of the link prediction task is to use the currently acquired network data (including structural information and attribute information) to predict which new connections will appear in the network. for the ogbl-citation dataset, the link prediction task is to predict the missing citations that are randomly dropped based on the existing edges. so the model is required to rank the missing references higher than the other existing references. according to this, we choose mrr as the evaluation metric [14]. besides, the validation set and testing set consist of randomly dropped edges. paradigmplus (2021) 2:3 improving the robustness of graphsaint via stability training 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . naturally, the training set consists of all the rest of the edges. table 1: results for the traditional graphsaint on the citation dataset of ogb graphsaint training validation test official 0.8626±0.0046 0.7933±0.0046 0.7943±0.0043 convergence 0.8690 0.8031 0.8048 non-convergence 0.0010 0.0010 0.0010 the results of the traditional graphsaint on the citation dataset of ogb are given in table 1. the official result of graphsaint is convergent: the mrr value of the training set is 0.8626 ±0.0046, the mrr value of the validation set is 0.7933 ±0.0046, and the mrr value of the test set is 0.7943 ±0.0043. the official mrr results are the statistical result of 10 runs. therefore, for the convenience of comparison, the hyperparameter named run will also be set to 10 in the following experiments. the bottom two rows in table 1 are the results of our recurrence experiment of the traditional graphsaint. we found that the traditional graphsaint will not converge in training with a probability range from 0.1 to 0.4 in the training process. for a run where the loss converges, the mrr results are consistent with the official results as shown in table 1, i.e., the training result can reach about 0.8690, the validation result can reach about 0.8031, and the test result can also reach about 0.8048. the loss curve for a run where the loss converges is shown in figure 1(a). we can see that one run contains 200 epochs. besides, in figure 1(a), as the epoch increases, the values of the loss gradually decrease overall, although the values of the loss fluctuate slightly in the process of the decline. the mrr results for a run where the loss does not converge are shown in the last row of table 1, i.e., the training result, the validation result, and the test result are all approximately equal to 0, which means that the mrr results of the run where the loss does not converge are invalid. and the loss curve for a run where the loss does not converge is shown in figure 1(b). we can see that after the 78th epoch, the loss is suddenly and sharply increased to 34.5388 and remains unchanged, which indicates that the loss can not converge afterward. therefore, some measures need to be taken to help the loss converge and obtain the valid mrr results for the link prediction task. to deal with the problem of the non-convergence in the training process, three different strategies, which are ln, bn, and flag, are applied to improve the traditional graphsaint. the results for the improved graphsaints on the citation dataset of ogb are given in table 2. in particular, we can see that there are four different improved graphsaints. in addition to the three strategies mentioned above, the results of a new combination strategy named bn+flag are also shown in the last row of table 2. as mentioned in section 3, both ln and bn belong to the normalization techniques, while flag belongs to the gnn tricks. and we are curious if there are better results for the improved graphsaints after applying the combination of the two types of strategies, that is bn and flag. therefore, the related experiments are added to verify our conjecture. in the following, the results of the four improved graphsaints as shown in table 2 will be explained in turn. besides, for each subfigure in figure 2, a blue dotted line, which is also the loss curve during training of the traditional graphsaint with convergence as shown in figure 1(a), is added. this blue dotted line is used to compare with the loss curve during training of the four improved graphsaints. at first, ln is listed in table 2 as an improved strategy for the traditional graphsaint, and the three mrr values are shown afterward: the training result can reach 0.8526±0.0151, the validation result can reach 0.7790±0.0113, and the test result can also reach 0.7791±0.0114. after applying the improved graphsaint with ln, the loss curve during training is shown by the solid red line in figure 2(a), where the loss curve is convergent. moreover, compared with the blue dotted line in figure 2(a), the solid red line fluctuates more sharply in the first 20 rounds at the beginning of training, but it will be more stable in the later period. in conclusion, we can see that although the improved graphsaint with ln has a loss of around 0.015 loss in accuracy compared with the official results of the traditional paradigmplus (2021) 2:3 8 yuying wang et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0 50 100 150 200 epoch 0 0.2 0.4 0.6 0.8 l o ss (a) with convergence 0 50 100 150 200 epoch 0 10 20 30 35 l o ss (b) without convergence figure 1: the loss curve during training of the traditional graphsaint table 2: results for the improved graphsaints on the citation dataset of ogb graphsaint training validation test ln 0.8526±0.0151 0.7790±0.0113 0.7791±0.0114 bn 0.9001±0.0014 0.8335±0.0020 0.8344±0.0023 flag 0.8880±0.0019 0.8165±0.0022 0.8175±0.0022 bn+flag 0.8997±0.0013 0.8349±0.0010 0.8364±0.0012 graphsaint, it solves the instability problem in the process of training. then, another normalization technique named bn is listed next to ln in table 2. compared with the official results for the traditional graphsaint, all the three mrr values have an improvement as shown in table 2: the training result can reach 0.9001±0.0014, the validation result can reach 0.8335±0.0020, and the test result can also reach 0.8344±0.0023. besides, the loss curve of the improved graphsaint with bn during training is shown by the solid red line in figure 2(b). we can see that the solid loss curve is convergent. besides, compared with the blue dotted line in figure 2(b), the improved graphsaint with bn has a stable convergence during training and a faster convergence rate. moreover, because the training time of the improved one with bn is shorter than that of the traditional graphsaint, the epoch is 150, as can be seen from the solid red line in figure 2(b), which is less than 200. here, it is also clear that the improved graphsaint with bn has a better test accuracy than the improved graphsaint with ln. the next row of table 2 is the experimental results of the improved graphsaint with flag. although the improvement is less than the improved graphsaint with bn, all the three mrr values still have an improvement compared with the official results for the traditional graphsaint as shown in table 2: the training result can reach 0.8880±0.0019, the validation result can reach 0.8165±0.0022, and the test result can also reach 0.8175±0.0022. besides, the loss curve of the improved graphsaint with flag during training is shown by the solid red line in figure 2(c). compared with the blue dotted line in figure 2(c), the improved graphsaint with flag also has a stable convergence in the training process, whose convergence rate is slightly slower than the improved one with bn. however, since the application of flag in the traditional graphsaint makes the training time longer, the paradigmplus (2021) 2:3 improving the robustness of graphsaint via stability training 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0 50 100 150 200 epoch 0 0.2 0.4 0.6 0.8 l o ss (a) ln 0 50 100 150 200 epoch 0 0.2 0.4 0.6 0.8 l o ss (b) bn 0 50 100 150 200 250 epoch 0 0.2 0.4 0.6 0.8 l o ss (c) flag 0 50 100 150 200 epoch 0 0.2 0.4 0.6 0.8 l o ss (d) bn+flag figure 2: the loss curve during training of the improved graphsaints epoch is 250, as can be seen from the solid red line in figure 2(c). the last row of table 2 is the results of the improved graphsaint with a new combination strategy named bn+flag. considering that both bn and flag bring an increase in the test accuracy to the traditional graphsaint, we conduct such an experiment by using the bn+flag combination strategy, and we can see that the training result can reach 0.8997±0.0013, the validation result can reach 0.8349±0.0010, and the test result can also reach 0.8364±0.0012. it’s a pity that the improved graphsaint with the bn+flag combination strategy has almost the same result of the test accuracy as the improved graphsaint with bn, which means that the flag strategy hardly improves the test accuracy of the results obtained by the improved graphsaint with bn. besides, the loss curve of the improved graphsaint with the bn+flag combination strategy during training is shown by the solid red line in figure 2(d). although the application of flag does not improve the test accuracy of the results, when compared with the blue dotted line in figure 2(d), we also find that the improved paradigmplus (2021) 2:3 10 yuying wang et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . graphsaint with the combination strategy achieves a faster convergence rate than the other three improved graphsaints. thus, the effectiveness of our improved graphsaints with the three strategies, that is ln, bn, and flag, are all verified. 5 conclusions the stability problem in the training process of gnn is crucial. in this paper, we focus on this stability problem and propose the improved graphsaints by applying the two normalization techniques, that is bn and ln, and a gnn trick named flag. experiments show that all the proposed algorithms improve the robustness of the training process of the traditional graphsaint and solve the problem of instability during training. besides, the improved algorithms with bn and flag can obtain a better test accuracy. moreover, the improved algorithms that apply bn also accelerate the convergence of the models. in the future, more attention will be paid to the stability of the gnn models in the process of distributed training for large graph datasets. acknowledgments this research was funded by the fifth innovative project of state key laboratory of computer architecture (ict, cas) under grant no. carch 5403. authors’ information – yuying wang received her ph.d. from the chinese academy of sciences. she is a postdoc at the institute of computing technology, chinese academy of sciences, beijing. – huixuan chi received his bachelor’s degree from north china electric power university. he is studying for a master’s degree at the institute of computing technology, chinese academy of sciences, beijing. – qinfen hao received his ph.d. from the chinese academy of sciences. he is a professor at the institute of computing technology, chinese academy of sciences, beijing. authors’ contributions – yuying wang participated in planning the manuscript outline, conducting the experiments, and preparing the manuscript. – huixuan chi participated in the results analysis and the manuscript preparation. – qinfen hao participated in the manuscript preparation. competing interests the authors declare that they have no competing interests. funding no funding was received for this project. paradigmplus (2021) 2:3 improving the robustness of graphsaint via stability training 11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . references [1] f. scarselli, m. gori, a. c. tsoi, m. hagenbuchner, and g. monfardini, “the graph neural network model,” ieee transactions on neural networks, vol. 20, no. 1, pp. 61–80, 2009. available at https://ieeexplore.ieee.org/abstract/document/4700287. [2] d. liben-nowell and j. kleinberg, “the link prediction problem for social networks,” in proceedings of the twelfth international conference on information and knowledge management, cikm ’03, p. 556–559, 2003. [3] j. chen, t. ma, and c. xiao, “fastgcn: fast learning with graph convolutional networks via importance sampling,” in international conference on learning representations, iclr ’18, p. 1–15, 2018. [4] j. lim, s. ryu, k. park, y. j. choe, j. ham, and w. y. kim, “predicting drug–target interaction using a novel graph neural network with 3d struc-ture-embedded graph representation,” journal of chemical information and modeling, vol. 59, no. 9, p. 3981–3988, 2019. available at https://pubs. acs.org/doi/abs/10.1021/acs.jcim.9b00387. [5] w. fan, y. ma, q. li, y. he, e. zhao, j. tang, and d. yin, “graph neural networks for social recommendation,” in the world wide web conference, www ’19, (new york, ny, usa), p. 417–426, 2019. [6] s. wu, y. tang, y. zhu, l. wang, x. xie, and t. tan, “session-based recommendation with graph neural networks,” in proceedings of the aaai conference on artificial intelligence, aaai ’19, p. 346–353, 2019. [7] z. wu, s. pan, f. chen, g. long, c. zhang, and p. s. yu, “a comprehensive survey on graph neural networks,” ieee transactions on neural networks and learning systems, vol. 32, no. 1, pp. 4– 24, 2021. [8] t. n. kipf and m. welling, “semi-supervised classification with graph convolutional networks,” in proceedings of the 5th international conference on learning representations, iclr ’17, pp. 1–14, 2017. [9] m. defferrard, x. bresson, and p. vandergheynst, “convolutional neural networks on graphs with fast localized spectral filtering,” in proceedings of the 30th international conference on neural information processing systems, nips ’16, p. 3844–3852, 2016. [10] h. zeng, h. zhou, a. srivastava, r. kannan, and v. prasanna, “graphsaint: graph sampling based inductive learning method,” in international conference on learning representations, iclr ’20, pp. 1–19, 2020. [11] w. l. hamilton, r. ying, and j. leskovec, “inductive representation learning on large graphs,” in proceedings of the 31st international conference on neural information processing systems, nips ’17, p. 1025–1035, 2017. [12] l. lü and t. zhou, “link prediction in complex networks: a survey,” physica a: statistical mechanics and its applications, vol. 390, no. 6, pp. 1150–1170, 2011. available at https://www. sciencedirect.com/science/article/abs/pii/s037843711000991x. [13] m. zhang and y. chen, “link prediction based on graph neural networks,” in advances in neural information processing systems, neurips ’18, p. 5165–5175, 2018. paradigmplus (2021) 2:3 https://ieeexplore.ieee.org/abstract/document/4700287 https://pubs.acs.org/doi/abs/10.1021/acs.jcim.9b00387 https://pubs.acs.org/doi/abs/10.1021/acs.jcim.9b00387 https://www.sciencedirect.com/science/article/abs/pii/s037843711000991x https://www.sciencedirect.com/science/article/abs/pii/s037843711000991x 12 yuying wang et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [14] w. hu, m. fey, m. zitnik, y. dong, h. ren, b. liu, m. catasta, and j. leskovec, “open graph benchmark: datasets for machine learning on graphs,” in advances in neural information processing systems, neurips ’20, pp. 22118–22133, 2020. [15] i. sutskever, j. martens, g. dahl, and g. hinton, “on the importance of initialization and momentum in deep learning,” in proceedings of the 30th international conference on machine learning, pmlr ’13, pp. 1139–1147, 2013. [16] j. duchi, e. hazan, and y. singer, “adaptive subgradient methods for online learning and stochastic optimization,” journal of machine learning research, vol. 12, no. 61, pp. 2121–2159, 2011. available at https://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf. [17] s. ioffe and c. szegedy, “batch normalization: accelerating deep network training by reducing internal covariate shift,” in proceedings of the 32nd international conference on international conference on machine learning, icml ’15, p. 448–456, 2015. [18] h. shimodaira, “improving predictive inference under covariate shift by weighting the loglikelihood function,” journal of statistical planning and inference, vol. 90, no. 2, pp. 227– 244, 2000. available at https://www.sciencedirect.com/science/article/abs/pii/ s0378375800001154. [19] n. srivastava, g. hinton, a. krizhevsky, i. sutskever, and r. salakhutdinov, “dropout: a simple way to prevent neural networks from overfitting,” the journal of machine learning research, vol. 15, no. 1, p. 1929–1958, 2014. [20] j. l. ba, j. r. kiros, and g. e. hinton, “layer normalization,” 2016. available at https://arxiv. org/abs/1607.06450. [21] y. wang, j. jin, w. zhang, y. yu, z. zhang, and d. wipf, “bag of tricks for node classification with graph neural networks,” 2021. available at https://arxiv.org/abs/2103.13355. [22] h. chi, y. wang, q. hao, and h. xia, “residual network and embedding usage: new tricks of node classification with graph convolutional networks,” 2021. available at https://arxiv. org/abs/2105.08330. [23] s. misra, “a step by step guide for choosing project topics and writing research papers in ict related disciplines,” in international conference on information and communication technology and applications, icta ’20, pp. 727–744, 2020. [24] h. shimodaira, “improving predictive inference under covariate shift by weighting the loglikelihood function,” journal of statistical planning and inference, vol. 90, no. 2, pp. 227–244, 2000. [25] t. raiko, h. valpola, and y. lecun, “deep learning made easier by linear transformations in perceptrons,” in proceedings of the fifteenth international conference on artificial intelligence and statistics, pmlr ’12, pp. 924–932, 2012. [26] k. kong, g. li, m. ding, z. wu, c. zhu, b. ghanem, g. taylor, and t. goldstein, “flag: adversarial data augmentation for graph neural networks,” 2020. available at https://arxiv.org/ abs/2010.09891. [27] a. madry, a. makelov, l. schmidt, d. tsipras, and a. vladu, “towards deep learning models resistant to adversarial attacks,” 2019. available at https://arxiv.org/abs/1706.06083. [28] y. wang and q. hao, “towards more robust gnn training with graph normalization for graphsaint,” applied informatics. icai 2021. communications in computer and information science., vol. 1455, pp. 82–93, 2021. paradigmplus (2021) 2:3 https://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf https://www.sciencedirect.com/science/article/abs/pii/s0378375800001154 https://www.sciencedirect.com/science/article/abs/pii/s0378375800001154 https://arxiv.org/abs/1607.06450 https://arxiv.org/abs/1607.06450 https://arxiv.org/abs/2103.13355 https://arxiv.org/abs/2105.08330 https://arxiv.org/abs/2105.08330 https://arxiv.org/abs/2010.09891 https://arxiv.org/abs/2010.09891 https://arxiv.org/abs/1706.06083 improving the robustness of graphsaint via stability training 13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [29] k. wang, z. shen, c. huang, c.-h. wu, y. dong, and a. kanakia, “microsoft academic graph: when experts are not enough,” quantitative science studies, vol. 1, no. 1, pp. 396–413, 2020. paradigmplus (2021) 2:3 introduction related work the sampling strategy of the traditional graphsaint the typical normalization techniques free large-scale adversarial augmentation on graphs (flag) methodology the improved graphsaints with the normalization techniques the improved graphsaint with a gnn trick experiments conclusions vol. 3, no. 3, pp 1 15, 2022 doi: 10.55969/paradigmplus.v3n3a1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . comparative analysis of machine learning techniques for the prediction of employee performance jide kehinde adeniyi1, abidemi emmanuel adeniyi id 1,2, yetunde josephine oguns id 3, gabriel olumide egbedokun id 3, kehinde douglas ajagbe id 4, princewill chima obuzor id 5, and sunday adeola ajagbe id 6,� 1landmark university, nigeria adeniyi.jide@lmu.edu.ng, adeniyi.emmanuel@lmu.edu.ng 2precious cornerstone university, nigeria 3the polytechnic ibadan, nigeria oguns.yetunde@polyibadan.edu.ng, g_egbedokun@yahoo.co.uk 4kogi state college of education, nigeria dougajagbe@kscoeankpa.edu.ng 5university of salford, united kingdom p.obuzor@edu.salford.ac.uk 6ladoke akintola university of technology lautech, nigeria saajagbe@pgschool.lautech.edu.ng abstract human resources’ purpose is to assign the best people to the right job at the right time, train and qualify them, and provide evaluation methods to track their performance and safeguard employees’ perspective skills. these data are crucial for decision-makers, but collecting the best and most useful information from such large amounts of data is tough. human resource employees no longer need to manually handle vast amounts of data with the advent of data mining. data mining’s primary goal is to uncover information hidden in data patterns and trends in order to produce results that are close to ideal. this study aims at comparing the performance of three techniques in the prediction of performance. the dataset undergoes preprocessing steps that include data cleaning, and data compression using principal component analysis. after preprocessing, training and classification were done using artificial neural network, random forest, and decision tree algorithm. the result showed that artificial neural networks performed the best in the prediction of employee performance keywords: machine learning ⋅ performance ⋅ human resources ⋅ employee ⋅ data mining received: 2 september 2022 ⋅ accepted: 10 november 2022 ⋅ published: 28 november 2022. 1 introduction data mining is a new but promising technique that analyses data from several angles and synthesizes it into useful knowledge. data mining’s purpose is to gain knowledge from existing data and this article is distributed under the terms of the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. https://doi.org/10.55969/paradigmplus.v3n3a1 https://orcid.org/0000-0002-2728-0116 https://orcid.org/0000-0001-7976-9258 https://orcid.org/0000-0003-4796-0503 https://orcid.org/0000-0002-8452-5917 https://orcid.org/0000-0002-9893-1473 https://orcid.org/0000-0002-7010-5540 2 jide kehinde adeniyi et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . improve the process. in the past, companies used manual appraisal for the evaluation of employees’ performance [1, 2]. employee performance is how an employee fulfills their tasks. typically, the appraisal process begins with a report from each department head, who offers recommendations for personnel advancement under his supervision [3]. the manual appraisal had some shortcomings: inconsistent reports, lack of standards, and delay in decision-making, which led to storing employees’ information in the database. the advanced data collection tools and database technology led to large amounts of data in the database. this massive increase in available data has had a significant impact on intelligent decision-making technology in recent years. as a result, people encounter an issue known as data drowning when they attempt to extract knowledge from data. however, human resources personnel no longer need to manually handle enormous amounts of data because there are machine learning techniques that can intelligently do this [4, 5]. machine learning is an artificial intelligence technique that is particularly important in prediction systems [6]. supervised, unsupervised, and semi-supervised machine learning approaches are the three main categories. of the three types, only the supervised use a pre-trained model for prediction. it creates a model using training data [7, 8]. to make the prediction, a training algorithm (machine learning) is used to develop a model. the machine learning algorithm trained the system on a tiny subset of the data before putting it to the test on the remainder. before making predictions, machine learning techniques can be applied to a sample of test data [4]. this study proposes the use of random forest (rf), artificial neural network (ann), and decision tree (dt) for the prediction of employee performance when given the job. the purpose is to assess how well each of the three approaches predicts an individual’s performance in a specific role. this is targeted at assisting the organization in deciding who to be given a task. in the remaining part of this research, section 2 describes the review of comparable works, and section 3 describes the content and methodology of the remaining portion of this study. section 4 contains the results and discussion, and section 5 contained the research’s conclusion. 2 related works several employee performance systems have been proposed in the literature. amongst the recent is the work of [9]. in their study, data mining techniques such as j48, naive bayes, and logistic regression are used to determine which workers will exit the business. multiple predictor values, as well as other key factors such as the number of tasks completed, supervisor assessment score, and expertise, are included in the details. they show that j48 operates well, with an accuracy of 98.84%. the authors of [10] used sentiment analysis on twitter to ascertain a person’s personality. support vector machine (svm), naive bayes (nb), and xgbosst classifiers were employed in the study to estimate a twitter user’s personality. their report showed an accuracy of 78%, 80%, and 85% for nb, svm, and xgboost respectively. xgboost was noted as having the best performance. the study in [11] proposed the use of machine learning to select the right player for cricket sport. several machine-learning approaches were considered for this. they include the support vector machine, decision tree, naive bayes classifiers, and random forest. they aim to solve player selection for a game of cricket using machine learning techniques. in the study of [12], the authors proposed the use of machine learning techniques to analyze the stroke dataset obtained from kaggle for stroke prediction. machine learning methods used include linear discriminant analysis, gaussian naive bayes, logistic regression, k-nearest neighbor classifier, support vector machine, random forest classifier, baffing classifier, gradient boosting classifier, and ada-boost classifier. the result of the comparison showed the best accuracy of 95.10% for the random forest algorithm. the study in [13] examined risk prediction in diabetes. in their paper, they attempted to predict the diabetic risk of an individual using several categories of examination data. statistical methods paradigmplus (2022) 3:3 comparative analysis of machine learning techniques for the prediction of employee performance 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . were used to examine the three different types of physical examination data: demographic information, vital signs, and laboratory results. an extreme gradient boosting (xgboost) model was utilized to predict diabetes and non-diabetic people. the system produced an accuracy of 0.95 with fasting blood glucose (fbg) and an accuracy of 0.77 without fbg. the authors in [14] worked on the evaluation of the quality of an employee. the proposed system collects and preprocesses employee performance data. it then uses this to predict the employee’s quality score using the backward propagation neural network (bpnn). the performance of bpnn was compared with the fuzzy neural network (fnn). it was concluded that while bpnn was excellent for adaptive and non-linear approximations, its efficiency dropped when the input variable has a high dimension. the fnn also easily falls into the local optimum, making it difficult to locate the global optimum. the study in [15] performed a comparison between three classification methods. the methods were compared to the ability to predict employee performance in a two-class scenario. the classification method compared includes logistic regression, decision tree, and naive bayes methods. it was observed that logistic regression gave a higher accuracy of 0.834 when compared with the decision tree and naive bayes methods (that gave 0.796 and 0.804 respectively). 3 material and methods in this study, a comparison of three classification methods for employee performance prediction is presented. an employee dataset obtained from kaggle was used as the input to the system. figure 1 depicts the proposed system’s overall structure. figure 1: block diagram of the system. paradigmplus (2022) 3:3 4 jide kehinde adeniyi et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 2: importing data into google collaboratory integrated development area. 3.1 dataset this study uses the human resource dataset obtained from the kaggle repository. employee data with a range of attributes make up the dataset. the csv is based on a fictitious business. names, dates of birth, marital status, gender, date of hire, department, grounds for termination, active or terminated status, position title, salary rate, manager name, and performance score are all included in the data collection. the dataset contains 36 attributes, each with 311 distinct values and no missing values [16]. the human resource dataset was loaded into google collaboratory ide. figure 2 shows the loaded dataset comprises 36 attributes with 311 unique values and no missing values. the target value is the performance attribute. this attribute has an employee that fully meets or exceeds the performance and those that do not meet the required performance (noted as pip needing performance improvement plan). the view of the table on the paper is limited. the performance column could not be shown in the screenshot captured. 3.2 data preprocessing this stage seeks to convert unstructured data into a form that machine learning systems can use. the steps include data cleaning/encoding and dimension reduction using principal component analysis (pca). 3.3 data cleaning/encoding a preliminary analysis of the dataset’s data instances and properties shows that preprocessing is needed since attributes are of different types (binary, numerical and nominal). since the classification methods will be operating on numeric data, non-numeric data are classified (or transformed) to a numeric value [1, 17]. 3.4 principal component analysis principal component analysis was also applied as a preprocessing step. by condensing a high number of variables into a smaller group that retains the majority of the data from the larger collection, paradigmplus (2022) 3:3 comparative analysis of machine learning techniques for the prediction of employee performance 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . principal component analysis (pca), a dimensionality reduction approach, decreases the dimensionality of large data sets. the key concept behind pca is to execute a linear mapping of data from a high-dimensional domain to a lower-dimensional space. the data’s variance is maximized [18]. the algorithm creates a new set of attributes by combining the existing ones. for formalization purposes, let x = x1, x2, .., xn be the dataset, in which each xi refers to a data instance. an instance xi described by d attributes is defined by the feature vector x1,1, .., xi,d. the following are some key points to remember about pca steps: – center the data by subtracting the values of each data instance xi by the mean µ according to zi = xi − µ – knowing that z = z1, z2, .., zn compute the covariance matrix using ∑= z t z – compute the eigenvalues δi = δ1, .., δd and the eigenvectors v of the covariance matrix using spectral decomposition that is presented ∑= v av −1, where a is a diagonal matrix with eigenvalues on the diagonal and zeroes elsewhere, and v is the matrix of eigenvectors. the eigenvalues on the diagonal of a correspond to the columns in v so that the first element of a is λ1, and the associated eigenvector is the first column of v , and so on. in order of decreasing difficulty, there is a need to select the eigenvalues and the k eigenvectors associated with the k biggest eigenvalues, where k is the reduced space’s number of dimensions (low-dimensional space). the primary components, which depict a linear transition from the initial attribute space to a new space with uncorrelated attributes, are defined by the eigenvectors. this is expressed by p c1 = c1,1x1+c1,2x2+..+c1,d xd in which p c1 denotes the l−th principal component (pc), x1, .., xd are the data attributes and c1,1, .., c1,d refer to the coefficients of p c1. the outputs from pca, i.e., the eigenvalues, the principal components, and their coefficients are useful for analyzing patterns in data. in many education data mining (edm) tasks, the identification of the main components affecting students’ performances is essential. as the original representation of the data (original attributes) was transformed into principal components, an analysis of the coefficients of principal components and the amount of explained variance was to obtain implicit knowledge from educational data. such coefficients express the correlation of each variable to the principal component, and its signal and magnitude are taken into account for interpreting the data patterns [18]. principal component analysis, a data pre-processing technique, was used to compress the dataset while retaining relevant information. several columns were observed to have no significance to the model, so it is dropped from the dataset. some of the columns dropped include employee_name, empid, salary, positionid, position, state, zip, dob, sex, date of hire, date of termination. out of the 36 attributes fed as features into the pca, 20 attributes retained the principal components. the dataset’s principal component of the test and train sets are shown in figure 3. 3.5 classification algorithms the algorithms compared for classification in this study include decision trees, random forests, and artificial neural networks. 3.5.1 decision tree the decision tree is a supervised machine learning technique that is non-parametric. a particular kind of target variable that is widely used in the classification of issues is a decision tree [19, 20]. it can operate in categorical or continuous modes for input and output variables. when using decision trees to address the prediction problem, attribute and class labels are represented by the external node and leaf node of the tree, respectively [20]. according to [21], assuming s is the training sample set and ∣s∣ is the number of samples contained in the training sample test. the sample is separated paradigmplus (2022) 3:3 6 jide kehinde adeniyi et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 3: principal component of the training and testing dataset. into n classes ∣c1∣, ∣c2∣, .., ∣cn∣. the probability of the sample s being of the class c is obtained using equation 1. p(si)= ∣ci∣ ∣s∣ (1) taking a as the number of attribute values, this can be given as x(a) as a set. we mark as sv the subset sample with value v. the entropy of the node sample set sv classification is given as e(s) if the branch node is chosen after attribute a is selected. to get the expected entropy value caused by a. the weighted sum of the entropy of each subset sv is calculated. the entropy can thus be given as presented in equation 2. entropy(s, a)=∑ ∣sv∣∣s∣ ∗ entropy(sv) (2) the information gain value gain(s, a) for the original sample set, s of attribute a is given in equation 3. gain(s, a)= entropy(s)− entropy(s, a) (3) where gain(s, a) is the compression of entropy expected as a result of the attribute selection of a. the more information is provided by the choice of test characteristic a for classification, the higher the gain(s, a). paradigmplus (2022) 3:3 comparative analysis of machine learning techniques for the prediction of employee performance 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5.2 random forest with a minor tweak to its settings, the widely used machine learning algorithm random forest is said to produce good results [22, 23, 24]. it is frequently used in classification because of its simplicity. numerous decision trees are in a random forest with various sample sets at each node [25]. to obtain an accurate result, the final score from each decision tree is averaged [26, 27]. as a result, a random forest is more reliable than a decision tree because it avoids bias and overfitting by randomly placing different trees in the training set [28]. suppose there is a training set x = x1, .., xn and responses y = y1, .., yn, bagging b times repeatedly selects a sample at random with the replacement of the training set and fits trees to the sample in [20]. 3.5.3 artificial neural network many real-world problems require using a neural network for processing, especially when developing a programming algorithm is challenging. the ann consists of neurons that act as input to the neural network (nn), hidden layers, and output layers. weights are added for the connection of the neurons [29, 30]. the ann is trained by showing examples and modifying the weight values of the network according to specified learning rules until the ann output matches the intended result [31]. a popular ann is the multilayer perceptron (mlp) and it is made up of neurons termed perceptron [32]. mlp is mathematically represented in equation 4 [33, 34, 35]. y = f ( n ∑ z=1 mz xz + b)s (4) where y is the output, xz is input vector z = 1, .., n, f is transfer function, mz is the weight vector, and b is bias. the minimized global error e using the training algorithm is given in equation 5. e = 1 n n ∑ n=1 en (5) taking n as the number of training patterns, en is the error corresponding to the training pattern n. en is represented in equation 6. en = 1 2 n ∑ q=1 (og − tg) (6) where n is the total output nodes, g is the gth output node, og is the network output at the gth output node, tg is the target output at the gth output node. in the mlp classifier, a relu activation function was used. there were 10 hidden layers in the architecture of the classifier and the mlp was trained to pick the information randomly. the maximum number of iterations was kept at 1000. 3.6 performance metrics in this study, the classification algorithms were evaluated and compared using the following metrics: accuracy, specificity, sensitivity, precision, f1-score, and matthews correlation coefficient. the matrices are obtained using true positive (tp), true negative (tn), false positive (fp), and false negative (fn). the tp is when the model properly predicts a positive class. the tn) is when the model properly predicts the negative class. the fp is when the model forecasts a positive class wrongly and the fn is when the model forecasts a negative class erroneously. accuracy is measured in classification tasks as the proportion of correct predictions made by the model out of all guesses, as presented in equation 7 [13, 36, 37]. paradigmplus (2022) 3:3 8 jide kehinde adeniyi et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . accuracy = t p + t n t p + f p + f n + t n (7) the statistic used to assess a model’s capacity to forecast true negatives in each accessible category is called specificity, which is expressed mathematically in equation 8 [35]. specif icity = t n f p + t n (8) recall (also called sensitivity, true positive rate, or tpr) is the ratio of the number of positive instances identified correctly to the number of instances with a positive class. it is obtained using equation 9 [6, 37, 38]. recall = t p t p + f n (9) precision (given in equation 10) is a metric that indicates the percentage of correct predictions made compared to the total number of forecasts made [36]. p recision = t p t p + f p (10) f1-score is the balanced average of the recall and precision of the classifier. it is calculated using equation 11 [18, 39]. f 1 = 2t p 2t p + f p + f n (11) 4 results and discussion in this section, the output of each major step of the system is examined. the output of the evaluation of the classification algorithms is also stated and compared with each other. a final comparison is made with other similar systems. for each algorithm, we divided the pre-processed dataset into a testing and training set. the dataset is divided into two halves, with 80 percent used for training and 20 percent used for testing. the training dataset is used to determine (or learn) the best variable combinations for building a strong predictive model. the testing dataset is used to evaluate the final model’s fit on the training dataset objectively. 4.1 decision tree classifier after the data was fed into the decision tree classifier, it gave an accuracy of 0.97. the accuracy, precision, specificity, sensitivity, and f1-score of the decision tree after testing are shown in table 1. figure 4 shows the confusion matrix. table 1: performance of evaluation performance measures (%) artificial neural network random forest decision tree accuracy 0.9894 0.9787 0.9681 precision 1.0000 1.0000 0.9867 specificity 1.0000 1.0000 0.9444 sensitivity 0.9868 0.9737 0.9737 f1 score 0.9934 0.9866 0.9801 paradigmplus (2022) 3:3 comparative analysis of machine learning techniques for the prediction of employee performance 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 4: confusion matrix classification using decision tree. 4.2 artificial neural network the type of neural network used was the multilayer perceptron. a feedforward artificial neural network called a multilayer perceptron produces a number of outputs from a set of inputs (mlp). the result of the neural network showed an accuracy of 0.99. the accuracy, precision, specificity, sensifigure 5: confusion matrix classification using artificial neural network. paradigmplus (2022) 3:3 10 jide kehinde adeniyi et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 6: confusion matrix classification using random forest. tivity, and f1-score of the decision tree after testing are shown in table 1. figure 5 shows the confusion matrix. 4.3 random forest when the dataset was fed into the random forest classifier, it had a classification accuracy of 0.97. the confusion matrix of the random forest classifier is shown in figure 6. the accuracy, precision, specificity, sensitivity, and f1-score of the decision tree after testing are shown in table 1. 4.4 comparison of decision tree and artificial neural network the roc curve shows how well the categorization thresholds performed. the real positive rate is plotted against the false positive rate on the curve. the receiver operating characteristics (roc) curve for neural network, decision tree, and random forest is shown in figure 7. in this study, table 1 shows the evaluation result and the ann outperformed others with an accuracy of 98.72%. also, although dt had better accuracy than rf, the specificity and precision of rf were observed to be higher than dt. in fact, in terms of specificity and precision, ann and rf have the same score. this shows that the ability of rf to predict true negatives is the same as that of ann. 5 conclusion and recommendation in this study, data mining techniques were used to construct a classification model for predicting employee performance using a real dataset from the kaggle repository. the decision tree (dt), artificial neural network, and random forest techniques were utilized to create the classification model and select the most relevant parameters that positively affect performance. it was observed after testing that although ann outperformed rf in terms of performance, they were evenly matched paradigmplus (2022) 3:3 comparative analysis of machine learning techniques for the prediction of employee performance 11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 7: roc curve for artificial neural network, decision tree, and random forest. in specificity and precision. this shows that the ability of ann and rf to accurately predict true negatives is the same. the performance of dt was lower than the other methods. this was expected as rf is a combination of several dts. however, due to the simplicity of dt, its performance was still impressive. among the three methods compared, ann would be recommended for performance prediction of employee performance in organizations with ample resources and in situations where critical decisions would be made on the result. in organizations with medium resources, rf would be recommended for use. in small organizations and in situations where the outcome of the predictions would not be critically used, dt is recommended. authors’ information – jide kehinde adeniyi received a b.sc. degree in computer science from adamawa state university, nigeria. he holds his m.tech. in computer science from the federal university of technology, nigeria. he obtained his doctoral degree in the department of computer science, university of ilorin, nigeria. his interest range various topics in biometrics, computer vision, security, artificial intelligence, and machine learning – abidemi emmanuel adeniyi is currently a lecturer and researcher in the department of computer sciences at precious cornerstone university, nigeria. his area of research interest is information security, the computational complexity of algorithms, the internet of things, and machine learning. he has published quite a number of research articles in reputable journal outlets. – yetunde josephine oguns is currently a ph.d. student at ajayi crowther university, nigeria. she obtained her master’s degree in computer science from the university of ibadan, and her bachelor of technology (b.tech.) in computer science from the federal university of technology akure, nigeria. she also obtained higher national diploma and national diploma in computer science respectively from computer science department, the polytechnic ibadan, nigeria, where she currently lectures. her research interest includes artificial intelligence, the paradigmplus (2022) 3:3 12 jide kehinde adeniyi et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . internet of things, data science, and smart solutions. she is a member of the computer professional association of nigeria (cpn), the nigeria computer society (ncs), the international association of engineers (iaeng), and the organization for women in science for the developing world (owsd). – gabriel olumide egbedokun is a ph.d. student in computer science at ajayi crowther university nigeria. he obtained m.sc and b.sc. in computer science from ajayi crowther university and the university of ibadan respectively. his research interests include data science, natural language processing, modeling, and the internet of things – kehinde douglas ajagbe is a lecturer in the department of computer science, kogi state college of education, nigeria. he currently serves as the head of the department of computer science and deputy director of the college ict centre. he earned a bachelor of technology degree (b.tech.) in computer science from the federal university of technology, nigeria in 2001. in 2013, obtained postgraduate diploma in education (pgde) from benue state university makurdi, nigeria. he received masters of science (m.sc) degree in computer science from the university of ilorin, nigeria in 2014. his research interests are in distributed computing, iot, and machine learning. he has many publications in highly rated databases. – princewill chima obuzor is a ph.d. candidate in data mining at the university of salford in the manchester united kingdom. he obtained a master’s in databases and web based systems at the university of salford and a b.sc. with distinction at v. dahl east ukrainian national university in luhansk, ukraine. his research interests are in machine learning and artificial intelligence, data science, process mining, and blockchain technologies. he is a member of the british computer society (bcs) in the uk. – sunday adeola ajagbe is a ph.d. candidate in the computer engineering department at the ladoke akintola university of technology (lautech), nigeria. he obtained m.sc and b.sc in information technology and communication technology respectively at the national open university of nigeria (noun), and he also has pgd in electronics and electrical engineering at lautech. he has earlier obtained hnd and nd in electrical and electronics engineering at the polytechnic, ibadan, ibadan, nigeria. his specialization includes artificial intelligence (ai), natural language processing (nlp), information security, data science, the internet of things (iot), biomedical engineering, and smart solutions. he is licensed by the council regulating engineering in nigeria (coren) as a professional electrical engineer. he is a member of the international association of engineers (iaeng), and the nigeria computer society (ncs). he is a student member of the institute of electrical and electronics engineers (ieee). he has published quite a number of research articles in reputable journal outlets such as springer, sciencedirect, ieee, iet press, and taylor & francis press. authors’ contributions – jide kehinde adeniyi participated in conceptualization, review, validation, writing of the original draft, and supervision. – abidemi emmanuel adeniyi participated in methodology, software, and coding. – yetunde josephine oguns participated in the literature search and writing of the original draft. – gabriel olumide egbedokun participated in the literature search. – kehinde douglas ajagbe participated in the literature search and writing of the original draft. – princewill chima obuzor participated in the software, project administration, and coding. – sunday adeola ajagbe participated in review editing and supervision. paradigmplus (2022) 3:3 comparative analysis of machine learning techniques for the prediction of employee performance 13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . competing interests the authors declare that they have no competing interests. funding no funding was received for this project. references [1] x. li, h. liu, w. wang, y. zheng, h. lv, and z. lv, “big data analysis of the internet of things in the digital twins of smart city based on deep learning,” future generation computer systems, vol. 128, pp. 167–177, 2022. [2] k. srinath, “page ranking algorithms–a comparison,” international research journal of engineering and technology (irjet), 2017. [3] m. okoye-ubaka, a. adewole, o. folorunso, and j. ezike, “neural network model for performance evaluation of academic staff of tertiary institutions,” international journal of applied information systems (ijais), 2013. [4] w. li, y. chai, f. khan, s. r. u. jan, s. verma, v. g. menon, x. li, et al., “a comprehensive survey on machine learning-based big data analytics for iot-enabled smart healthcare system,” mobile networks and applications, vol. 26, no. 1, pp. 234–252, 2021. [5] n. seliya, a. abdollah zadeh, and t. m. khoshgoftaar, “a literature review on one-class classification and its potential applications in big data,” journal of big data, vol. 8, no. 1, pp. 1–31, 2021. [6] m. yağcı, “educational data mining: prediction of students’ academic performance using machine learning algorithms,” smart learning environments, vol. 9, no. 1, pp. 1–19, 2022. [7] y. k. atri, s. pramanick, v. goyal, and t. chakraborty, “see, hear, read: leveraging multimodality with guided attention for abstractive text summarization,” knowledge-based systems, vol. 227, p. 107152, 2021. [8] s. uddin, a. khan, m. e. hossain, and m. a. moni, “comparing different supervised machine learning algorithms for disease prediction,” bmc medical informatics and decision making, vol. 19, no. 1, pp. 1–16, 2019. [9] z. jaffar, w. noor, and z. kanwal, “predictive human resource analytics using data mining classification techniques,” int. j. comput, vol. 32, no. 1, pp. 9–20, 2019. [10] k. a. nisha, u. kulsum, s. rahman, m. hossain, p. chakraborty, t. choudhury, et al., “a comparative analysis of machine learning approaches in personality prediction using mbti,” in computational intelligence in pattern recognition, pp. 13–23, springer, 2022. [11] c. kapadiya, a. shah, k. adhvaryu, and p. barot, “intelligent cricket team selection by predicting individual players’ performance using efficient machine learning technique,” int. j. eng. adv. technol, vol. 9, no. 3, pp. 3406–3409, 2020. [12] d. paikaray and a. k. mehta, “an extensive approach towards heart stroke prediction using machine learning with ensemble classifier,” in proceedings of the international conference on paradigms of communication, computing and data sciences, pp. 767–777, springer, 2022. paradigmplus (2022) 3:3 14 jide kehinde adeniyi et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [13] h. yang, y. luo, x. ren, m. wu, x. he, b. peng, k. deng, d. yan, h. tang, and h. lin, “risk prediction of diabetes: big data mining with fusion of multifarious physical examination indicators,” information fusion, vol. 75, pp. 140–149, 2021. [14] t.-y. tseng and q. luo, “company employee quality evaluation model based on bp neural network,” journal of intelligent & fuzzy systems, vol. 40, no. 4, pp. 5883–5892, 2021. [15] m. g. t. li, m. lazo, a. k. balan, and j. de goma, “employee performance prediction using different supervised classifiers,” 2021. [16] d. rich, “human resources data set.” https://www.kaggle.com/datasets/rhuebner/ human-resources-data-set?select=hrdataset_v14.csv, 2021. [17] m. fathi, m. haghi kashani, s. m. jameii, and e. mahdipour, “big data analytics in weather forecasting: a systematic review,” archives of computational methods in engineering, pp. 1–29, 2021. [18] v. r. p. borges, s. esteves, p. de nardi araújo, l. c. de oliveira, and m. holanda, “using principal component analysis to support students’ performance prediction and data analysis,” in brazilian symposium on computers in education (simpósio brasileiro de informática na educação-sbie), vol. 29, p. 1383, 2018. [19] k. annisa and a. arpan, “decision tree algorithm and gini calculation for puppies skin disease diagnosis,” jurnal mantik, vol. 5, no. 4, pp. 2682–2687, 2022. [20] l. e. vivanco-benavides, c. l. martínez-gonzález, c. mercado-zúñiga, and c. torres-torres, “machine learning and materials informatics approaches in the analysis of physical properties of carbon nanotubes: a review,” computational materials science, vol. 201, p. 110939, 2022. [21] h. liu and j. liu, “female employment data analysis based on decision tree algorithm and association rule analysis method,” scientific programming, vol. 2022, 2022. [22] m. lotfirad, h. esmaeili-gisavandani, and a. adib, “drought monitoring and prediction using spi, spei, and random forest model in various climates of iran,” journal of water and climate change, vol. 13, no. 2, pp. 383–406, 2022. [23] a. abdulhafedh, “comparison between common statistical modeling techniques used in research, including: discriminant analysis vs logistic regression, ridge regression vs lasso, and decision tree vs random forest,” open access library journal, vol. 9, no. 2, pp. 1–19, 2022. [24] h. valecha, a. varma, i. khare, a. sachdeva, and m. goyal, “prediction of consumer behaviour using random forest algorithm,” in 2018 5th ieee uttar pradesh section international conference on electrical, electronics and computer engineering (upcon), pp. 1–6, ieee, 2018. [25] d. rajković, a. marjanović jeromela, l. pezo, b. lončar, f. zanetti, a. monti, and a. kondić špika, “yield and quality prediction of winter rapeseed—artificial neural network and random forest models,” agronomy, vol. 12, no. 1, p. 58, 2021. [26] f. hidayat and t. m. s. astsauri, “applied random forest for parameter sensitivity of low salinity water injection (lswi) implementation on carbonate reservoir,” alexandria engineering journal, vol. 61, no. 3, pp. 2408–2417, 2022. [27] s. dharumarajan and r. hegde, “digital mapping of soil texture classes using random forest classification algorithm,” soil use and management, vol. 38, no. 1, pp. 135–149, 2022. paradigmplus (2022) 3:3 https://www.kaggle.com/datasets/rhuebner/human-resources-data-set?select=hrdataset_v14.csv https://www.kaggle.com/datasets/rhuebner/human-resources-data-set?select=hrdataset_v14.csv comparative analysis of machine learning techniques for the prediction of employee performance 15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [28] a. t. prihatno, h. nurcahyanto, and y. m. jang, “predictive maintenance of relative humidity using random forest method,” in 2021 international conference on artificial intelligence in information and communication (icaiic), pp. 497–499, ieee, 2021. [29] a. k. jain, j. mao, and k. m. mohiuddin, “artificial neural networks: a tutorial,” computer, vol. 29, no. 3, pp. 31–44, 1996. [30] a. j. kehinde, a. e. adeniyi, r. o. ogundokun, h. gupta, and s. misra, “prediction of students’ performance with artificial neural network using demographic traits,” in recent innovations in computing, pp. 613–624, springer, 2022. [31] s. wang, j. di, d. wang, x. dai, y. hua, x. gao, a. zheng, and j. gao, “state-of-the-art review of artificial neural networks to predict, characterize and optimize pharmaceutical formulation,” pharmaceutics, vol. 14, no. 1, p. 183, 2022. [32] a. duykuluoglu, “the significance of artificial neural networks in educational research a summary of research and literature,” technium biochemmed, vol. 2, no. 2, pp. 107–116, 2021. [33] s. w. chin, k. g. tay, c. c. chew, a. huong, and r. a. rahim, “dorsal hand vein authentication system using artificial neural network,” indonesian journal of electrical engineering and computer science, vol. 21, no. 3, pp. 1837–1846, 2021. [34] r. tabbussum and a. q. dar, “performance evaluation of artificial intelligence paradigms—artificial neural networks, fuzzy logic, and adaptive neuro-fuzzy inference system for flood prediction,” environmental science and pollution research, vol. 28, no. 20, pp. 25265–25282, 2021. [35] g. piecuch and r. żyła, “diagnosing extrusion process based on displacement signal and simple decision tree classifier,” sensors, vol. 22, no. 1, p. 379, 2022. [36] y. zhang, p. geng, c. sivaparthipan, and b. a. muthu, “big data and artificial intelligence based early risk warning system of fire hazard for smart cities,” sustainable energy technologies and assessments, vol. 45, p. 100986, 2021. [37] s. a. ajagbe, j. b. awotunde, m. a. oladipupo, and o. e. oye, “prediction and forecasting of coronavirus cases using artificial intelligence algorithm,” in machine learning for critical internet of medical things, pp. 31–54, springer, 2022. [38] o. d. adeniji, d. b. adekeye, s. a. ajagbe, a. o. adesina, y. j. oguns, and m. a. oladipupo, “development of ddos attack detection approach in software defined network using support vector machine classifier,” in pervasive computing and social networking, pp. 319–331, springer, 2023. [39] e. o. ogunseye, c. a. adenusi, a. c. nwanakwaugwu, s. a. ajagbe, and s. o. akinola, “predictive analysis of mental health conditions using adaboost algorithm,” paradigmplus, vol. 3, no. 2, pp. 11–26, 2022. paradigmplus (2022) 3:3 introduction related works material and methods dataset data preprocessing data cleaning/encoding principal component analysis classification algorithms decision tree random forest artificial neural network performance metrics results and discussion decision tree classifier artificial neural network random forest comparison of decision tree and artificial neural network conclusion and recommendation vol. 1, no. 2, pp 1 17, 2020 doi: 10.55969/paradigmplus.v1n2a1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . using analog ensembles with alternative metrics for hindcasting with multistations carlos balsa id 1,4 �, c. veiga rodrigues id 2, isabel lopes id 1,3, and josé rufino id 1,4 1instituto politécnico de bragança (ipb), campus de santa apolónia, 5300-253 bragança, portugal {balsa, isalopes, rufino}@ipb.pt 2vestas wind systems a/s design center porto, portugal carlos.rodrigues@fe.up.pt 3applied management research unit (uniag), 4research centre in digitalization and intelligent robotics (cedri) abstract this study concerns making weather predictions for a location where no data is available, using meteorological datasets from nearby stations. the hindcast with multiple stations is performed with different variants of the analog ensemble (anen) method. in addition to the traditional monache metric used to identify analogs in datasets from one or two stations, several new metrics are explored, namely cosine similarity, normalization, and k-means clustering. these were analyzed and benchmarked to find the ones that bring improvements. the best results were obtained with the k-means metric, yielding between 3% and 30% of lower quadratic error when compared against the monache metric. also, by making the predictors to include two stations, the performance of the hindcast improved, decreasing the error up to 16%, depending on the correlation between the predictor stations. keywords: analog ensembles ⋅ metrics ⋅ hindcasting ⋅ time series ⋅ meteorological data. received: 18 may 2020 ⋅ accepted: 14 june 2020 ⋅ published: 19 june 2020. 1 introduction the idea of weather prediction using analog ensembles is not new. it was described as early as 1969, by lorenz [1] who concluded that such a method would not work. however further works managed to prove the usefulness of this approach in a much more limited scope, in fields ranging from meteorology to flood study, especially thanks to the decisive contributions of van den dool [2, 3]. the analog ensemble (anen) method enables to improve the accuracy of a deterministic numerical weather prediction (nwp) model that makes a forecast. alongside this forecast, historical forecasts are also available (a record of forecasts from nwp at past dates), as well as historical observations (a record of real meteorological data observed in a given place). then, to improve forecasting accuracy, the forecast is compared to historical forecasts. the historical forecasts analogs to the current one are kept, and the observations corresponding to the dates of the best analogs are used to improve the forecast value. the name of the method comes from there: past forecasts close to the current one are called analogs, and these are used to obtain an ensemble of past observations. this article is distributed under the terms of the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. https://doi.org/10.55969/paradigmplus.v1n2a1 https://orcid.org/0000-0003-2431-8665 https://orcid.org/0000-0002-9030-7158 https://orcid.org/0000-0002-5614-3516 https://orcid.org/0000-0002-1344-8264 2 carlos balsa et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . the anen method is mostly used as a post-processing procedure. a deterministic nwp forecast model reports on a possible state of the atmosphere among many possible states. as the predicted state does not always match reality (due to the limitations of the model and its inputs), it is useful to have a probability distribution function (pdf) of the possible states that quantifies the uncertainty associated with the prediction. the forecast pdf can be estimated using a set of n past validation observations corresponding to the n best analogs (past model forecasts) to the current nwp model forecast [4]. as the anen method allows us to quantify the uncertainty associated with the forecast of a meteorological variable, this method was also used to generate probabilistic wind power forecasts [5]. in the field of meteorology, a significant contribution to the use of the analog ensemble method was made by monache et al., [6] having been refined [4] and applied to a variety of operational scenarios [5, 7, 8], where its accuracy and usefulness were clearly demonstrated. since the anen method provides a prediction based on past observations, it results in an efficient downscaling procedure that eliminates representativeness issues. these arise from the fact that observations and model output can be associated with quantities averaged over different spatiotemporal scales [6]. this article is an extended version of previous work [9]. it focuses on the problem of making weather predictions for a given location where no forecast is available, using meteorological datasets from nearby stations. with this aim, the anen method was applied to a hindcasting problem, where the time-series of a meteorological variable in a given location was reconstructed using data from neighbor weather stations. to discover ways to improve on the original anen method, alternative metrics to determine the analogs were considered, and their performance was compared, allowing us to find out which ones exhibit the best accuracy. to investigate the impact of more than one predictor station on the performance of the hincasting, this evaluation was conducted both with a single predictor station dataset, and also using the datasets from two stations. the rest of this paper is structured as follows. section 2 presents the anen methodology, the mathematical formulation of all the metrics and prediction methods considered, and the definitions for error quantification. section 3 describes the field data used in this study, the numerical results produced by the computational execution of the anen methodology and an analysis of those results. section 4 concludes with final remarks and lays out directions for future work. 2 the analog ensembls methodology the analog ensemble method is relatively simple. having an accurate similarity metric is crucial, though, for the success of the forecasting. this section presents alternative metrics and prediction methods that can be used for this purpose. it also describes the tools used to evaluate the performance of the different metrics. 2.1 applying the analog ensemble approach the classical anen method builds on an observed dataset consisting of real records of meteorological variables, and on a historical dataset that corresponds to a time-series from a numerical weather prediction (nwp) model used to forecast [4, 5] or hindcast [10] meteorological data. figure 1 illustrates how the analog ensemble (anen) method is used in this work, to predict the values of a weather station using another station. comparing this scenario with the classical application of the anen method, the dataset from the predictor station corresponds to the historical dataset, and the dataset from the station to be predicted corresponds to the observation data set; moreover, real measurements from meteorological stations were used as the historical dataset. historical and observation datasets must exist over a training period. the larger this period, the better the anen method performs [11]. the other period considered is the prediction period (or paradigmplus (2020) 1:2 using analog ensembles with alternative metrics for hindcasting with multistations 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . futurepast predictor station historical dataset: 1 futurepast predicted station observation dataset: predicted value 2 3 training period prediction period figure 1: the analog ensemble method (adapted from [4]). reconstruction period in an hindcasting context), in which only data from the predictor station is available. as depicted in figure 1, the anen procedure is implemented in three steps for each time step to predict or reconstruct. firstly, in step 1 the values analog to the prediction are obtained by scanning the historical dataset from the predictor station. analogs are past occurrences deemed close enough to the current prediction, classified as such according to an analog metric. step 2 consists of matching these analogs with the corresponding observations at the target station. step 3 consists of estimating the target value with the past values matched to the analogs (e.g., weighted average). 2.2 metrics used to find analogs in the present work various metrics were used to assess and compute the similarity between meteorological data of the predictor station. by analogy with the original anen method, the data from the predictor station included in the prediction period will be referred to as forecasts. the first metric used was the one originally established by monache [4] for the anen method. it will be simply referred to as the monache metric from now on. this metric is based on the euclidean distance: it computes the difference between the values of atmospheric variables in two windows of time. one of the windows is the forecast and the other belongs to the preceding historical dataset. the monache metric is given by equation 1 and the meaning of its terms is shown in table 1. nv ∑ i=1 wi σfi ¿ á á áà k ∑ j=−k (fi,t+j − ai,t′+j)2 (1) considering fi,t and ai,t′ as two vectors of size 2k + 1, equation 1 may be rewritten simply as presented in equation 2. m1 = nv ∑ i=1 wi σfi ∣∣fi,t − ai,t′∣∣, (2) where ∣∣.∣∣ represents the euclidean norm. an alternative metric is obtained by normalising fi,t as well as ai,t′ in equation 2, resulting in equation 3. m2 = nv ∑ i=1 wi σfi ∥ fi,t ∣∣fi,t∣∣ − ai,t′ ∣∣ai,t′∣∣ ∥ . (3) paradigmplus (2020) 1:2 4 carlos balsa et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 1: terms of the monache metric. ft forecast at given time t in the prediction period. at′ analog forecast at a time t′ in the training period. nv number of meteorological variables taken into account when comparing forecasts. wi weight given to variable i. σfi standard deviation of variable i in the historical dataset. k integer equal to half the width of the time window over which the metric is computed. fi,t+j value of the forecast at time t + j in the time window for a given variable i. ai,t′+j value of the analog at time t′ + j in the time window for a given variable i. normalized vectors all have a norm of 1. the idea behind this reasoning is to look at the global weather pattern present in both forecasts. it can be seen that the basic monache metric looks not only for a similar weather pattern, but also for similar numerical values to the various variables used in the forecast. as a consequence, it will not keep as an analog a forecast behaving exactly like the forecast to improve, but at higher or lower values. normalization aims at solving this perceived problem. this method is called normalized monache, hereinafter abbreviated to normalized. another approach is to consider the cosine of the angle θ between vectors fi,t and ai,t′, given equation 4. cos(θ) = ati,t′fi,t ∥ai,t′∥∥fi,t∥ (4) where ati,t′ denotes the transpose of vector ai,t′. the cosine can then be used to estimate the analog by means of the correlations between the two vectors, as demonstrated by k. adachi [12] as presented in equation 5. m3 = nv ∑ i=1 wi σfi cos(θ). (5) it is known that this metric behaves like the correlation coefficient, taking values between -1 and 1 (1 = maximum similarity). the idea of the cosine metric is thus to replace the monache metric with the cosine of the angle between the forecast and the analog, keeping the analogs with cosine closest to 1. lastly, clustering can also be applied to this problem. clustering is the partitioning of data into clusters of similar data. in this case each 2k+1 vector ai,t′ is assigned to a similar cluster. each cluster constitutes an analog ensemble. clustering the historical dataset produces several analog ensembles that can then be used immediately. the only task left is to assign the current forecasting to the good analog ensemble (i.e., to the closest cluster). this method was inspired by gutierrez et al., [13] who used clustering in a forecasting problem. the technique used to cluster the historical dataset is the k-means [14]. this method is considered an optimization method because it seeks a partition that minimizes the sum of the distance between all the vectors in the same cluster (see for instance [15]). from now on, this method is mentioned as k-means and the corresponding metric as m4. 2.3 prediction methods once the best na analog have been identified, by means of one of the metrics described in the previous section, the prediction of the meteorological variable at time t can then be made. this is accomplished by means of the observations values (oti , with i = 1, 2, . . . ,na) at time steps matching the analogs (see figure 1). the predicted value is then the average of these observation values, as given by equation paradigmplus (2020) 1:2 using analog ensembles with alternative metrics for hindcasting with multistations 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6. pt = 1 na na ∑ i=1 oti. (6) equation 6 needs to be adjusted when the normalized monache (m2) or cosine (m2) metrics are used to obtain the analog ensemble. this is because in these cases what is looked at are trends and not exact equal numerical values, and so there might be a difference between the value of a meteorological variable at time t′ and the desired value. equation 6 thus becomes equation 7, where δtti = ft − ati accounts for the scale difference between the value of the variable in the analog and in the forecast. pt = 1 na na ∑ i=1 (oti + δtti). (7) to improve the accuracy of the prediction, it is tempting to use data coming from various weather stations as predictors, instead of data coming from just one station. this raises the problem of how to treat this additional data. this problem was solved in two different ways, which were both used in this paper to determine which method is more adequate to handle data coming from various stations. the first approach is the dependent stations method. this considers the stations to be nothing more than additional predictor variables, and as such computes analogs across all stations at once every time. that it, the observation at time t′ is deemed to be an analog of the weather at time t if, and only if, the weather at all stations at time t′ is close to the weather at all stations at time t. a second approach is the independent stations variant, in which the metric is calculated at each station independently from the other. the prediction is then made using the mean of the analogs from all the stations. compared to the first approach, each station now forms a disjoint set of data in which analogs are searched separately. then, weights are assigned to each stations to form the final set of analogs; for instance, with two independent stations, 90% of analogs may come from the study of the data from one station and the remaining 10% will come from the study of the data from the other station. with α representing the weight of analogs from predictor station 1 and β = 1 −α standing for the weight of analogs from predictor station 2, the predicted value will be given by equation 8. pt = 1 na ⎛ ⎝ αna ∑ i=1 ot1i + βna ∑ j=1 ot2j ⎞ ⎠ (8) where ot1i and ot2i are the observations that match the analogs from the predictors station 1 and 2. 2.4 error assessment measuring the errors of predictions against truth values (the values effectively registered by weather stations) is of paramount importance in order to assess the performance of the prediction methods. as shown by chai and draxler [16], assessing a model accuracy is best done using various metrics. three metrics are especially useful when trying to assess the performance of a forecasting model. the simplest of those metrics is the bias, given by equation 9. bias = 1 n n ∑ i=1 (xi − yi), (9) where n is the number of predictions, xi is a prediction and yi is the corresponding truth value. as its name suggests, the bias measures the bias of the model, which is simply the average error compared to the truth. however, it does not really show the behavior of the error. it is useful to determine if the model makes predictions that are lower or higher than the truth, but it is not enough in itself to really know how well the model performs. it only shows the systematic error of the model. paradigmplus (2020) 1:2 6 carlos balsa et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . thus, complementary to the bias, the root mean-squared error (rmse) is also used and corresponds to equation 10. rmse = ¿ á áà 1 n n ∑ i=1 (yi − xi)2. (10) the rmse is useful because the squared terms give a higher weight to higher errors. thus, the rmse will be higher if the model makes predictions which are far from the truth, even if these erroneous predictions are few in number. to complement the bias and rmse metrics, the standard deviation of the error (sde) may also be considered. the sde simply corresponds to equation 11. sde = √ rmse2 − bias2. (11) considering the bias as a basic indicator of the systematic error in a prediction, then the sde is the equivalent indicator of the random error. finally, another useful metric, whose usage alongside the rmse metric is recommended by chai and draxler [16], is the mean absolute error (mae) as presented in equation 12. mae = 1 n n ∑ i=1 ∣yi − xi∣. (12) basically, the mae metric computes an average distance to the truth, in absolute value. in turn, the bias, whose expression is similar to the mae minus the distance modulus, simply computes the average error in a way that allows for positive and negative errors to cancel each other. therefore, the mae gives a somewhat more truthful assessment of the average distance to the truth. a low bias and high mae means that the model is not really accurate, but that its predictions are sometimes higher than the truth, and sometimes lower. thus, taking the mae into consideration is necessary to really understand how the error is distributed in the forecast, since it also shows systematic error, but this time in terms of absolute distance. 3 computational experiments in this section the anen method enriched with the new similarity metrics, prediction methods and error assessment tools laid out previously, is put to the test, using real-world meteorological data. 3.1 meteorological datasets testing was done using the data from meteorological stations located on the coast of the state of virginia, usa. these stations were used because their observations are freely available from the united states national data buoy center [17]. the location of the stations is shown in figure 2. the data extends from the years 2012 to 2018. the data from the years 2012 to 2016 (training period) was kept as an historical database, and the aim of the experiment was to predict (reconstruct) the data from 2017 to 2018 (prediction/reconstruction period), at one station. the stations data was resampled by time-integration to match a sampling period of 6 minutes. these stations monitor 6 different meteorological variables: pressure (pres), air temperature (atmp), water temperature (wtmp), wind speed (wspd), gust speed (gst) and wind direction (wdir). often, time series are not complete, data is missing for periods whose length may be short (hours) or long (months). the idea was therefore to hindcast: at one station where only the meteorological data from 2012 to 2016 (training period) was known, the goal was to reconstruct, for that station, the data from 2017 to 2018 (prediction period), using other stations (predictors), where data for the full range of 2012 to 2018 was known. based on the anen procedure illustrated in figure 1, the data from 2012 to 2016 paradigmplus (2020) 1:2 using analog ensembles with alternative metrics for hindcasting with multistations 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 2: geolocation of the ndbc metheorological stations in viginia [17] at the predicted station is the observed dataset whilst the data at all other stations between 2012 to 2018 is the historical dataset (comprised of multiple time series); also, the training and prediction periods correspond to intervals 2012 to 2016, and 2017 to 2018, respectively. the main advantage of this setup is that it makes easy to evaluate the model accuracy, because one can compare the estimates obtained with the anen method with the real values. it is important to stress, however, that the data collected by stations every six minutes over a period of 7 years is huge, an so it would be very time consuming to sequentially process all the records in the historical dataset. instead, the problem was simplified to predict the weather between 10 am and noon, using analogs of the weather for that specific time period. this greatly reduced computing time, while still providing data from different years and different seasons, thus covering very different weather patterns. 3.2 tests results the results of the computational tests performed in the context of this work can be divided into two parts. in a first study, the importance of the choice of stations was assessed, to find how important choosing the right stations to hindcast the values at another station is. then, since it is also possible to assign weights to stations, the importance of choosing the right weight was also studied. in both studies all the four metrics were applied to obtain the analog ensemble. all the tests where performed with a half time window k = 20, which corresponds to a time window of four hours in length (2 hours before the forecast, and two hours after), and a dimension of the analog ensemble of na = 25. the numerical results next presented and discussed were produced by a computational application developed in r [18] and executed in a linux system of an hpc cluster located at cedri/ipb. paradigmplus (2020) 1:2 8 carlos balsa et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 assessment of the predictor station as already stated, the first study aimed to evaluate how important is the choice of the predictor stations. to evaluate this, the gust speed (gst) data for the ykt station (yktv2 in figure 2) between the years 2017 and 2018 was reconstructed using the value of gst at three different pairs of stations. table 2 contains results (errors for different metrics used to find analogs) for an anen hindcast whose predictor was based solely on the mnp station (mnpv2 in figure 2). figure 3 shows the positioning of the results in table 2, also including the standard deviation (sde) of each metric. the figure shows the same information present in table 2 but in a visual form, taking advantage of the relation between rmse, bias and sde in equation 11. by representing the bias and sde as coordinates, the rmse becomes the distance to the origin. if the bias is assumed as a zero-order measure of systematic error, then sde is a measure of the random error. hence, figure table 2: predicting gst at the ykt station using the mnp station as predictor metric bias (m/s) rmse (m/s) mae (m/s) m1 0.51 1.98 1.48 m2 0.38 2.05 1.54 m3 0.37 2.03 1.53 m4 0.39 1.93 1.44 1.0 0.5 0.0 0.5 1.0 1.5 bias (m/s) 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 sd e (m /s ) rmse = 0.5 m/s rmse = 1 m/s rmse = 1.5 m/s rmse = 2 m/s m1 m2 m3 m4 figure 3: predicting gst at ykt station using mnp station as predictor position of each metric in relation to errors and standard deviation. paradigmplus (2020) 1:2 using analog ensembles with alternative metrics for hindcasting with multistations 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 3: predicting gst at the ykt station using two different predictor stations metric station 1 station 2 bias (m/s) rmse (m/s) mae (m/s) m1 dom ykr 0.95 2.34 1.75 m1 dom mnp 0.94 2.34 1.74 m1 wds mnp 0.24 1.75 1.33 m2 dom ykr 0.26 1.79 1.36 m2 dom mnp 0.68 2.01 1.52 m2 wds mnp 0.49 1.83 1.43 m3 dom ykr 0.43 1.81 1.37 m3 dom mnp 0.87 2.12 1.61 m3 wds mnp 0.73 1.94 1.51 m4 dom ykr 0.30 1.63 1.22 m4 dom mnp 0.43 1.85 1.38 m4 wds mnp 0.24 1.69 1.28 3 gives insight on the nature of the prediction error. table 3 shows results on extending the anen method to include a second station as a predictor. pairs were made of stations mnp, dom, ykr and wds to assess results consistency across different pairs. 1.0 0.5 0.0 0.5 1.0 1.5 2.0 bias (m/s) 0.0 0.5 1.0 1.5 2.0 sd e (m /s ) rmse = 0.5 m/s rmse = 1 m/s rmse = 1.5 m/s rmse = 2 m/s m1 dom+ykr m2 dom+mnp m3 wds+mnp m4 figure 4: predicting gst at ykt station using two different predictor stations position of each option in relation to errors and standard deviation. paradigmplus (2020) 1:2 10 carlos balsa et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 4 shows the corresponding positioning of the results, again including the sde metric. compared to this extension, the results in table 2 and figure 3 show less error and less sde for the k-means (m4) metric, while retaining similar bias to the cosine (m3) and the normalized (m2) metric. as the main source of error is due to the sde, the error in the predictions is not due to a systematic offset of the anen techniques employed. this is in line with the results from table 3, where the k-means method consistently shows better performance. the simple application of the monache metric (m1) yielded higher bias (which is also consistent with the results in table 3). normalizing the monache metric shows overall improvements in the bias and rmse, though more evident in the results from table 3. it is only for the <wds, mnp > pair of predictors that the monache metric shows a superior performance, though the k-means metric still has lower rmse and mae. comparing results from table 2 and table 3, using mnp alone as predictor is worse than using mnp and either dom or wds for the k-means (m4) and normalized monache (m2) metrics. for the cosine (m3) and monache (m1)metrics, it is clearly better to use both wds and mnp instead of mnp alone for hindcasting. however, for these metrics, it is better to use mnp alone rather than both dom and mnp. comparing the pure monache metric (m1) with the normalized one (m2), the results in table 3 show that the latter leads to bias and rmse reduction. the only predictor pair where this was not observed was < wds, mnp >; yet, the difference was not meaningful as it corresponds to 4% of higher rmse. the cosine metric (m3) behavior resembles the normalized metric, though with degraded performances in the error metrics. this similarity was expected as both methods look solely for similar relative patterns in the time series. as it can be seen in table 3 and figure 4, k-means (m4) behave in the same way as normalized monache (m2) and cosine (m3), which implies that clusterization employs a similar idea as these two methods. results, however, are noticeably better. these results show a rift between the methods: while k-means (m4), cosine (m3) and normalized monache (m2) all give results following the same trend, monache’s results go in another direction. there is, however, a rational explanation for this behavior: monache looks for analogs by minimizing the distance between the value of the target variable at time t (when the prediction is made) and at time t′ (the analog). the other methods, however, disregard this distance. instead, they look for a similar evolution of the weather during the time window. as such, the results implies that at the station dom weather follows a similar pattern as the station ykt, but because of the different location, meteorological values are not the same. this difference in value disturbs the monache metric, but not the others, who look at the underlying weather patterns. all methods, however, give their worse results with the <dom, mnp > pair, and always by a clear margin, while the <wds, mnp > pair performs well in all cases. this suggests that wds is a much better station to predict the weather at ykt compared to dom, and that ykr is a very good predictor station too, since it is able to offset the inaccuracies caused by the use of dom as predictor (except in the case of the monache method, since monache gives a great emphasis on numerical distance between values). overall, the <mnp , wds > pair is the one giving the lower rmse overall across all methods. figure 4 shows a trend where the error decreases mainly due to improvements in the bias (hence systematic error) rather than the sde. moreover, the <dom , mnp > pair of stations yield worse agreement due to higher bias. as expected, results are improved when using two stations, compared to using just one. however, the results for monache and cosine metrics suggest that the choice of stations is important to really have an accuracy gain. 3.2.2 assessment of the predictor stations weights once the importance of using the correct stations is assessed, it becomes important to evaluate if weighting the contribution from each predictor station can improve on the accuracy of the weather prevision at the ykt station. it is also important to compare the two ways of combining information from the two predictor stations, described in the section 2.3. for this, the focus is kept on the <wds, paradigmplus (2020) 1:2 using analog ensembles with alternative metrics for hindcasting with multistations 11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 4: predicting gst at station ykt using weighted predictor station metric α β bias (m/s) rmse (m/s) mae (m/s) m1 – – 0.24 1.75 1.33 m1 0.1 0.9 0.52 1.95 1.45 m1 0.5 0.5 0.40 1.78 1.33 m1 0.9 0.1 0.27 1.74 1.31 m2 – – 0.49 1.83 1.43 m2 0.1 0.9 0.61 1.99 1.55 m2 0.5 0.5 0.54 1.87 1.45 m2 0.9 0.1 0.49 1.83 1.42 m3 – – 0.73 1.94 1.51 m3 0.1 0.9 0.05 1.92 1.49 m3 0.5 0.5 0.24 1.95 1.51 m3 0.9 0.1 0.05 1.93 1.50 m4 – – 0.24 1.69 1.28 m4 0.1 0.9 1.22 2.56 1.95 m4 0.5 0.5 0.50 2.16 1.66 m4 0.9 0.1 -0.25 2.10 1.65 1.0 0.5 0.0 0.5 1.0 1.5 2.0 bias (m/s) 0.0 0.5 1.0 1.5 2.0 sd e (m /s ) rmse = 0.5 m/s rmse = 1 m/s rmse = 1.5 m/s rmse = 2 m/s m1 wds mnp m2 0.1wds 0.9mnp m3 0.5wds 0.5mnp m4 0.9wds 0.1mnp figure 5: predicting gst at station ykt using weighted predictor station position of each option in relation to errors and standard deviation. paradigmplus (2020) 1:2 12 carlos balsa et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . mnp> pair, whose results gave the lowest rmse overall in the previous test. the question is to determine if it is possible to improve these results even further by assigning weights to these stations. to this aim, both the independent stations and the dependent stations variants were used. the former variant allows weights to be set for each individual station, while the latter does not (recall section 2.3). in accordance with the equation 8, α is the weight of predictor station wds and β is the weight of predictor station mnp. as a consequence, in table 4, tests results showing "–" in the weight columns are tests ran with the dependent method, while tests with numerical values in the weight columns are tests ran with the independent method. the target variable was kept the same (gst), for ease of comparison with the previous results. all the results are in table 4 and figure 5. considering the results from table 4 and looking at the stations independently, the monache metric (m1) yields the best results, but only if the weights are equal. however setting most weight on wds (α) is better than doing it on mnp (β). the normalized monache metric (m2), however, prefers to have the analogs looked across all the stations at once. the cosine metric (m3) shows no big difference between the dependent and independent methods. the independent method performs slightly better when maximal weight is assigned to one station. in agreement with previous results, it appears that independent k-means metric gives best results when wds has most of the weight. however, even then it performs clearly worse than dependent k-means. 00 0.2-0.2 0.4-0.4 0.6-0.6 0.8-0.8 0.9-0.9 0.95-0.95 0.99-0.99 1-1 0. 460. 921. 41. 82. 32. 73. 23. 74. 14. 655. 566. 46. 97. 3 001.1 1.12.3 2.33.4 3.44.6 4.6 standard deviation sde correlation coefficient ● ● m1 m2 m3 m4 figure 6: a taylor diagram comparing the best of each of the four methods. the square at the bottom represents the truth values paradigmplus (2020) 1:2 using analog ensembles with alternative metrics for hindcasting with multistations 13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 5 does not show a clear trend, though it may be considered that improvements are mostly due to lower bias instead of sde. moreover, there are no changes in the sde when weights are swapped between predicting stations (e.g. 0.1/0.9 to 0.9/0.1), and for most methods there is no advantage in applying those weights. the exception is the k-means method (m4), which is sensitive to the weights as bias largely changes. the best results, however, may be obtained for the dependent stations, e.g. without weighting. the figure 6 presents the taylor diagram [19] for the best case of each method. it is possible to see that in general the normalized metric (m2) gives the best results. its proximity to the truth indicates a high correlation coefficient with it, a low sde value and similar positions on the horizontal axis shows similar standard deviations in other words, the prediction obtained by the normalized metric is close to the truth. predictions from the k-means (m4) and monache (m1) metrics are very close to one another, and also close to the truth, with a high correlation coefficient and a low sde value. however, they are closer to the origin on the horizontal axis, indicating that their standard deviation is lower than that of the truth. this implies these methods have troubles following the variations of gst accurately. in general cosine (m3) is the metric which performs the worst its coefficient correlation with the truth is lower than in other methods, and the rms distance to the truth is higher. however, it has the closest standard deviation to the truth, meaning the cosine method is the method which follows the variations of gst the most accurately. figure 7 shows the prediction with the different metrics jointly with the truth values. as expected, the forecast by the normalized metrics is the closest to the truth. it is interesting to note that while the prevision by the monache and k-means metrics behave similarly, they look rather different from one another. cosine metric, as expected, is the farthest from the truth but displays a lot of variability. 0 5 10 15 20 day g st (m /s ) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● mon. tue. wed. thu. fri. sat. sun. ● truth m1 m2 m3 m4 figure 7: prevision compared to the truth, for the first week of january 2018 paradigmplus (2020) 1:2 14 carlos balsa et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . qualitatively the cosine and normalized metrics give a representation of the time series with higher fidelity, due to additional variance. quantitatively, however, the additional variance introduces mismatches which result in poorer accuracy when compared to the k-means and monache metrics. 4 conclusion in this work, meteorological data was predicted at one location based on multiple historical datasets from weather stations located in other places. in order to achieve this prediction, the analog ensemble method was applied and several variants were explored by changing the metric used to determine the analogs in the historical dataset. the prediction period was two years, based on a training period of four years of historical and observed time series. from all these results, it appears that the choice of predictor stations, and how to weight them if a weighted approach is used, has a significant bearing on the hindcasting, and presumably forecasting, accuracy. the problem of selecting stations for hindcasting and forecasting purposes is a non-trivial one, and from these experiments, it would appear that the best way to make a viable selection is to test hindcasting on known data simply, to determine which are the stations most suited to forecasting and hindcasting purposes at the target location. the use of the k-means metrics leads to an improvement ranging from 3% to 30% of lower quadratic error when compared against the monache metric. increasing the predictors to two stations improved the performance of the hindcast, leading up to 16% of lower error, depending on the correlation between the predictor stations. these features show the kind of improvements which can be made on the existing anen method. the results look very promising. however, it is necessary to optimize several parameters that have not yet been analyzed. therefore, the number of clusters was left at a basic value. it is possible to tweak this value and see how to set this value for maximal accuracy, or even control the size of clusters. the k-means algorithm is also a reasonably basic clustering algorithm, and more accurate algorithms now exist. it would be interesting to look at their performances compared to the basic k-means in this case. it is also possible to look at larger scales, both in terms of the number of variables, number of stations, or distance between stations. the study presented in this paper focused on a rather simple testing environment, with stations located all next to each other. as a next step, it seems logical to look at the performances under larger scales of the various approaches studied. finally, the potential of the analog ensemble method to predict the weather variables at a given location can be useful to develop models for the operational management of agricultural plantations highly dependent on the weather, using short and medium-range weather forecasts downscaled to the spatial scales representative of the agricultural plantations location. authors’ information – carlos balsa studied numerical methods at the universidade do porto and at institut national polytechnique de toulouse, where he received a ph.d. in computational sciences and applied mathematics. currently, he teaches mathematics at the polytechnic institute of bragança and his research interests are the numerical and computational methods concerning fluid mechanics, heat transfer, and data mining. – c. veiga rodrigues holds a diploma course from the von karman institute for fluid dynamics and a ph.d. in mechanical engineering by university of porto. he is currently an engineer at vestas wind systems a/s, working in data analytics and atmospheric flow modeling focusing wind energy power plants. formerly, he was a researcher at universidade do porto, having worked in the coupling between numerical weather models and engineering flow solvers. paradigmplus (2020) 1:2 using analog ensembles with alternative metrics for hindcasting with multistations 15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . – isabel lopes holds a ph.d. in information systems and technologies by universidade do minho. she is adjunct professor at the instituto politécnico de bragança and a researcher with the applied management research unit (uniag) in the same institute. she is also a member of the algoritmi research center at universidade do minho, has more than 70 published papers, and belongs to several scientific journals’ editorial boards. – josé rufino holds a degree in systems and informatics engineering by universidade do minho, and a phd in informatics by the same university. he currently works as coordinator professor at the instituto politécnico de bragança, where he teaches computer engineering subjects and researches with the research centre in digitalization and intelligent robotics (cedri) in highperformance and parallel computing, with a focus on scientific computing problems. authors’ contributions – carlos balsa developed the new metrics and prediction methods presented in section 2, participated in the design of algorithms and its implementation in r, performed part of the tests presented in section 3, and wrote parts of all the sections of the manuscript. – c. veiga rodrigues imported and prepared the meteorological data sets used by the r code, validated the test results using the error assessment metrics presented in section 2.4, produced the graphs presented in section 3, and participated in the writing of all sections of the manuscript. – isabel lopes coordinated the writing of the manuscript and performed part of the tests presented in section 3.2. – josé rufino improved the computational performance of the r code by exploiting parallelization facilities of the r language, monitored the execution of the tests in the cedri hpc cluster, and participated in the writing of all sections of the manuscript. competing interests the authors declare that they have no competing interests. funding no funding was received for this project. references [1] e. n. lorenz, “atmospheric predictability as revealed by naturally occurring analogues,” journal of the atmospheric sciences, vol. 26, no. 4, pp. 636–646, 1969. https://journals.ametsoc.org/ view/journals/atsc/26/4/1520-0469_1969_26_636_aparbn_2_0_co_2.xml. [2] h. m. van den dool, “a new look at weather forecasting through analogues,” monthly weather review, vol. 117, no. 10, pp. 2230–2247, 1989. https://journals.ametsoc.org/view/journals/ mwre/117/10/1520-0493_1989_117_2230_anlawf_2_0_co_2.xml?tab_body=pdf. [3] h. van den dool, “searching for analogues, how long must we wait?,” tellus a: dynamic meteorology and oceanography, vol. 46, no. 3, pp. 314–324, 1994. https://onlinelibrary.wiley.com/ doi/abs/10.1034/j.1600-0870.1994.t01-2-00006.x. paradigmplus (2020) 1:2 https://journals.ametsoc.org/view/journals/atsc/26/4/1520-0469_1969_26_636_aparbn_2_0_co_2.xml https://journals.ametsoc.org/view/journals/atsc/26/4/1520-0469_1969_26_636_aparbn_2_0_co_2.xml https://journals.ametsoc.org/view/journals/mwre/117/10/1520-0493_1989_117_2230_anlawf_2_0_co_2.xml?tab_body=pdf https://journals.ametsoc.org/view/journals/mwre/117/10/1520-0493_1989_117_2230_anlawf_2_0_co_2.xml?tab_body=pdf https://onlinelibrary.wiley.com/doi/abs/10.1034/j.1600-0870.1994.t01-2-00006.x https://onlinelibrary.wiley.com/doi/abs/10.1034/j.1600-0870.1994.t01-2-00006.x 16 carlos balsa et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [4] l. delle monache, f. a. eckel, d. l. rife, b. nagarajan, and k. searight, “probabilistic weather prediction with an analog ensemble,” monthly weather review, vol. 141, no. 10, pp. 3498–3516, 2013. https://journals.ametsoc.org/view/journals/mwre/141/10/mwr-d-12-00281.1. xml. [5] s. alessandrini, l. delle monache, s. sperati, and j. nissen, “a novel application of an analog ensemble for short-term wind power forecasting,” renewable energy, vol. 76, pp. 768–781, 2015. https://www.sciencedirect.com/science/article/abs/pii/s0960148114007915. [6] l. delle monache, t. nipen, y. liu, g. roux, and r. stull, “kalman filter and analog schemes to postprocess numerical weather predictions,” monthly weather review, vol. 139, no. 11, pp. 3554– 3570, 2011. https://journals.ametsoc.org/view/journals/mwre/139/11/2011mwr3653.1. xml. [7] s. alessandrini, l. delle monache, s. sperati, and g. cervone, “an analog ensemble for shortterm probabilistic solar power forecast,” applied energy, vol. 157, pp. 95–110, 2015. https:// www.sciencedirect.com/science/article/abs/pii/s0306261915009368. [8] s. alessandrini, l. delle monache, c. m. rozoff, and w. e. lewis, “probabilistic prediction of tropical cyclone intensity with an analog ensemble,” monthly weather review, vol. 146, no. 6, pp. 1723–1744, 2018. https://journals.ametsoc.org/view/journals/mwre/146/6/ mwr-d-17-0314.1.xml. [9] a. chesneau, c. balsa, c. v. rodrigues, and i. m. lopes, “hindcasting with multistations using analog ensembles,” in ceur workshop proceedings, vol. 2486, pp. 215–229, ceur-ws, 2019. http://ceur-ws.org/vol-2486/icaiw_ikit_2.pdf. [10] e. vanvyve, l. delle monache, a. j. monaghan, and j. o. pinto, “wind resource estimates with an analog ensemble approach,” renewable energy, vol. 74, pp. 761–773, 2015. https://www. sciencedirect.com/science/article/abs/pii/s0960148114005308. [11] “national center for atmospheric research.” https://nar.ucar.edu/. [12] k. adachi, matrix-based introduction to multivariate data analysis. springer singapore, 2016. https://link.springer.com/book/10.1007/978-981-15-4103-2. [13] j. m. gutiérrez, a. s. cofiño, r. cano, and m. a. rodríguez, “clustering methods for statistical downscaling in short-range weather forecasts,” monthly weather review, vol. 132, no. 9, pp. 2169– 2183, 2004. https://journals.ametsoc.org/view/journals/mwre/132/9/1520-0493_2004_ 132_2169_cmfsdi_2.0.co_2.xml. [14] j. macqueen, “some methods for classification and analysis of multivariate observations,” in proceedings of the fifth berkeley symposium on mathematical statistics and probability, vol. 1, pp. 281– 297, oakland, ca, usa, 1967. [15] l. eldén, matrix methods in data mining and pattern recognition. philadelphia, pa, usa: siam, 2007. [16] t. chai and r. r. draxler, “root mean square error (rmse) or mean absolute error (mae)? – arguments against avoiding rmse in the literature,” geoscientific model development, vol. 7, no. 3, pp. 1247–1250, 2014. https://gmd.copernicus.org/articles/7/1247/2014/. [17] “national data buoy center.” https://www.ndbc.noaa.gov/. [18] r core team, r: a language and environment for statistical computing. r foundation for statistical computing, vienna, austria, 2017. https://www.yumpu.com/en/document/view/6853895/ r-a-language-and-environment-for-statistical-computing. paradigmplus (2020) 1:2 https://journals.ametsoc.org/view/journals/mwre/141/10/mwr-d-12-00281.1.xml https://journals.ametsoc.org/view/journals/mwre/141/10/mwr-d-12-00281.1.xml https://www.sciencedirect.com/science/article/abs/pii/s0960148114007915 https://journals.ametsoc.org/view/journals/mwre/139/11/2011mwr3653.1.xml https://journals.ametsoc.org/view/journals/mwre/139/11/2011mwr3653.1.xml https://www.sciencedirect.com/science/article/abs/pii/s0306261915009368 https://www.sciencedirect.com/science/article/abs/pii/s0306261915009368 https://journals.ametsoc.org/view/journals/mwre/146/6/mwr-d-17-0314.1.xml https://journals.ametsoc.org/view/journals/mwre/146/6/mwr-d-17-0314.1.xml http://ceur-ws.org/vol-2486/icaiw_ikit_2.pdf https://www.sciencedirect.com/science/article/abs/pii/s0960148114005308 https://www.sciencedirect.com/science/article/abs/pii/s0960148114005308 https://nar.ucar.edu/ https://link.springer.com/book/10.1007/978-981-15-4103-2 https://journals.ametsoc.org/view/journals/mwre/132/9/1520-0493_2004_132_2169_cmfsdi_2.0.co_2.xml https://journals.ametsoc.org/view/journals/mwre/132/9/1520-0493_2004_132_2169_cmfsdi_2.0.co_2.xml https://gmd.copernicus.org/articles/7/1247/2014/ https://www.ndbc.noaa.gov/ https://www.yumpu.com/en/document/view/6853895/r-a-language-and-environment-for-statistical-computing https://www.yumpu.com/en/document/view/6853895/r-a-language-and-environment-for-statistical-computing using analog ensembles with alternative metrics for hindcasting with multistations 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [19] k. e. taylor, “summarizing multiple aspects of model performance in a single diagram,” journal of geophysical research: atmospheres, vol. 106, no. d7, pp. 7183–7192, 2001. https://agupubs. onlinelibrary.wiley.com/doi/abs/10.1029/2000jd900719. paradigmplus (2020) 1:2 https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2000jd900719 https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2000jd900719 introduction the analog ensembls methodology applying the analog ensemble approach metrics used to find analogs prediction methods error assessment computational experiments meteorological datasets tests results assessment of the predictor station assessment of the predictor stations weights conclusion vol. 2, no. 2, pp 17 32, 2021 doi: 10.55969/paradigmplus.v2n2a2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . a blockchain-based approach to support an iso 9001:2015 quality management system rafael bettín-díaz id 1, camilo mejía-moncayo id 2, and alix e. rojas id 1,� 1universidad ean, bogotá, colombia {rbettind4339,aerojash}@universidadean.edu.co 2école de technologie supérieure, montreal, canada camilo.mejia-moncayo.1@ens.etsmtl.ca abstract quality is an essential element for any company that wants to be recognized successfully. the companies take part in a certification process under a quality standard, such as iso 9001 for varied reasons. some of them are to improve the performance of its operation, differentiate itself from its competitors, achieve a better position in the market, and export quickly. getting and maintaining such certification can put high pressure on the company. even, become a source of corruption risk in emerging markets, where companies may be tempted to perform unethical practices. such as falsifying or adulterating documents to maintain their certifications and the benefits derived from it. thus, it becomes necessary to reinforce with technology the quality management system audit process to minimize corruption risk. given this, here is the proposed software architecture for a quality management system supported by blockchain technology. this architecture helps to guarantee the integrity and immutability of the information, exposing any fraud attempt and even facilitates the audit process’s automation. keywords: software architecture ⋅ blockchain ⋅ iso 9001-2015 ⋅ quality management system received: 3 june 2021 ⋅ accepted: 21 august 2021 ⋅ published: 31 august 2021. 1 introduction iso 9001 is a standard of quality management systems (qms) with international recognition, as is shown by over one million companies and organizations in over 170 countries certified [1]. iso 9001 certification is not mandatory; however, it can be a useful tool to add credibility and differentiate an organization from its competitors [2]. the certification allows demonstrating that a company complies with standards and procedures that will provide a high level of customer service [3]. the primary motivation for iso 9001 certification is to improve the products/services, and most importantly, the process that produces them [4]. if the process is done correctly, iso standards can improve the business processes and add real value to the company, enhancing its performance[5, 6, 7, 8]. for obtaining an iso 9001 certification, an external body audits the company requesting the certification. this process includes verifying the control of the documentation, the records kept the conformity of the personnel and management with the system, the functioning of the system in each area of the organization, and the training of the personnel to comply with the requirements, among this article is distributed under the terms of the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. https://doi.org/10.55969/paradigmplus.v2n2a2 https://orcid.org/0000-0003-1938-190x https://orcid.org/0000-0002-3661-2231 https://orcid.org/0000-0002-0371-3925 18 rafael bettín-díaz et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . others [9]. however, suppose that the process is not well implemented. in that case, the company can not obtains the certificate or lost it. that produces continuous pressure on the company to maintain the certification, becoming a source of corruption risk. the company could be tempted to do unethical practices like paying a bribe, adulterating or falsifying registries, or other corrupted methods to keep the certification [10]. even, for some industries in emerging markets, quality certification is a contractual requirement to export to some markets[5]. in this context, the audit of information in repositories and databases has some drawbacks since the registries can be modified, added, or deleted by unethical practices or even cyber-attacks [11, 12]. all this makes it impossible to guarantee real transparency and reliability of the information to be used by certifying bodies. it could be almost impossible for an auditor to detect whether a document was tampered with or whether the evidence is the actual result of a well-done procedure. unless the system can maintain immutable records and allow traceability of the process. due to the aforementioned, in this study, a blockchain-based software architecture to guarantee the immutability, traceability, and transparency of the quality assurance and certification process under iso 9001:2015 is proposed. this architecture takes the essence of quality management systems, with their requirements and principles, to be reinforced by blockchain technology. to provide greater confidence and transparency in the information and documentation needed to carry out an iso 9001:2015 certification process that allows the audit of this information to be performed anywhere by any person with access to the blockchain and even facilitates its automation. 2 literature review this section explains the essential concepts regarding iso 9001:2015 quality management systems and blockchain technology. 2.1 quality management systems and iso 9001:2015 nowadays, there are many interpretations of what quality is, a simple definition is meeting customer requirements. for iso 9001:2015 quality management system[13], is the “degree to which a set of inherent characteristics of an object fulfills requirements”. the iso 9000, is a generic series of quality management standards for quality management systems (qms). a qms includes all activities of the overall management function that determine the quality policy, objectives, and responsibilities and their implementation. iso 9001:2015 apply to the business process within an organization and can be used by manufacturing or service industries [14]. iso 9001 is based on seven principles that represent a set of fundamental beliefs, norms, rules, and values as described iso [15], these principles are customer focus, leadership, engagement of people, process approach, improvement, evidence-based decision-making, and relationship management, table 1 provides a description of them. iso 9001 may be implemented in different ways depending on organization management, thanks to its flexibility and the scope for the certification process, which might explain the unmatched performance of these standardized systems [16]. the standard was the first released in 1987, since then, it has been a significant quality movement and framework for all kinds of organizations worldwide; it is estimated that more than one million companies have adopted this standard since its creation[17]. according to the american society for quality [18], iso 9001 is based on a plan-do-check-act methodology and provides a process-oriented approach to documenting and reviewing the structure, responsibilities, and procedures required to achieve effective quality management in an organization. specific sections of this standard contain information on topics such as requirements for a quality management system, including documented information, planning and determining process interactions; responsibilities of management; management of resources, including human resources and an organization’s work environment; product realization, including the steps from design to delivery; paradigmplus (2021) 2:2 a blockchain-based approach to support an iso 9001:2015 quality management system 19 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 1: iso quality management principles quality principles description customer focus sustained success is achieved when an organization attracts and retains the confidence of customers and other interested parties. every aspect of customer interaction provides an opportunity to create more value for the customer. understanding the current and future needs of customers and other interested parties contributes to the sustained success of the organization. leadership creation of the unity of purpose and direction and engagement of people enable an organization to align its strategies, policies, processes, and resources to achieve its objectives. engagement of people to manage an organization effectively and efficiently, it is important to involve all people at all levels and to respect them as individuals. recognition, empowerment, and enhancement of competence facilitate the engagement of people in achieving the organization’s quality objectives. process approach the quality management system consists of interrelated processes. understanding how results are produced by this system enables an organization to optimize the system and its performance. improvement improvement is essential for an organization to maintain current levels of performance, react to changes in its internal and external conditions, and create new opportunities. evidence-based decision-making decision making can be a complex process, and it always involves some uncertainty. it often involves multiple types and sources of inputs, as well as their interpretation, which can be subjective. it is important to understand cause-and-effect relationships and potential unintended consequences. facts, evidence, and data analysis lead to greater objectivity and confidence in decision-making. relationship management interested parties influence the performance of an organization. sustained success is more likely to be achieved when the organization manages relationships with all of its interested parties to optimize their impact on its performance. relationship management with its suppliers and partners networks is of particular importance. measurement, analysis, and improvement of the qms through activities like internal audits and corrective and preventive action [13]. where standardized processes allow guarantee the repeatability of the product or service quality features. the adoption of iso 9001 in a company requires an implementation process that will be involved all company statements. in this sense, the british assessment bureau [19], suggests the following steps for a successful implementation: 1○ make the right choice: make sure the standard it has chosen fits the organization’s needs; also make sure it already has processes that can be assessed. 2○ reviewing the standard: it will help it to learn the quality management system it wants to be certified. 3○ communicating the goal: the implementation process requires teamwork from all its collaborators. additional effort will be necessary; it is important to communicate the objectives and the target to be achieved. paradigmplus (2021) 2:2 20 rafael bettín-díaz et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4○ establish training requirements: get prepared to be ready. based on its team’s knowledge about the standard, could be beneficial to improve its skills to help the implementation process 5○ using a consultant: iso standards implementation requires a consultant to help companies get certified. it is not the consultant’s responsibility for implementing the iso standard. each company handles its implementation. senior management should be actively involved in the process to achieve the goal. 6○ select a certification body: iso implementations are based on a three-year cycle; make sure the certification body can provide it with the companionship it needs for this specific objective. 7○ developing a management system: these standards apply to any organization in any industry sector. the framework provided for this standard will not tell it how to manage processes in its organization; it has to implement a quality management system based on the standard that applies to its own organization in concordance with its resources, production, measurement, analysis, and improvements. 8○ the stage 1 audit: the process starts with a "stage 1 audit". the auditor will review the process and provide a gap analysis that will help to identify the actions required for meeting the standard. 9○ the stage 2 audit: once your organization has filled the gaps identified during the stage 1 audit, it comes to the stage 2 audit, will demonstrate the effectiveness of your quality management system and that your organization meets the requirement for certification. 10○ maintaining the management system: this can be the hardest part of the certification, it is the organization responsible for ensuring that itself applies the quality management system once the certification has been granted. communication and training will be necessary along the way, internal audits must be implemented to make sure the standards are being met, and corrective action should be held to assure quality management. once qms is implemented, an organization that requests iso 9001 certification should demonstrate that fulfills the requirements to be certified [20]. where the core of the certification process is the audit; in which the compliance standardized processes against the registries or document evidence provided by the company is verified. this process must be performed by an independent, impartial, and free from conflicts of interests certification body. depending on the audit findings, whether the company or process is certified or not. however, as was defined by the iso 9000 principles, the result depends on the information registered in the evidence. this system feature could represent a weak element of the system in some contexts. because the evidence could be changed or deleted for varied reasons, and the process will depend on the honesty of the participants (company and certification body). unfortunately, these situations happen although contradict the system principles. 2.2 blockchain in 2009, as the world was reeling from a meltdown in the financial sector and politicians were musing about what could and should be done, a project called bitcoin quietly dropped onto the global stage; satoshi nakamoto [21] published the article bitcoin: a peer-to-peer electronic cash system in october 2008, in which he described the cryptocurrency called bitcoin. many electronic cash schemes existed prior to bitcoin, but none of them achieved widespread use. by adopting blockchain technology, bitcoin achieved compelling capabilities that promoted its use. the use of a blockchain-enabled bitcoin to be implemented in a distributed fashion so that no single user controlled the currency and no single point of failure existed [22]. paradigmplus (2021) 2:2 a blockchain-based approach to support an iso 9001:2015 quality management system 21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . blockchain may have been used as the underlying technology for all bitcoin’s transactions, but its applicability goes beyond. this technology itself can be described as the general ledger for all transactions of any kind of assets ever made [23]. blockchain enables a potentially evolving and open set of parties to maintain a safe, permanent, and tamper-proof digital ledger of transactions, without a central authority. the key to the technology is that transactions are not recorded centrally; instead, each party maintains a copy of the ledger. most parties need to approve (verify) a new transaction before it can be recorded in the ledger – according to a notion of the majority that varies depending on the specific technology. once a transaction is approved, it is almost impossible to change it or remove it. hence, blockchain technology can be seen as a replicated append-only transactional data store, and hence it can be used as a substitute for centralized registers maintained by single trusted authorities [24], it has the potential to revolutionize the digital world by enabling a distributed consensus where each and every online transaction involving digital assets, past and present, can be verified at any time in the future. it does this without compromising the privacy of the digital assets and parties involved. distributed consensus and anonymity are two important characteristics of blockchain technology. blockchain 2.0, goes beyond cryptocurrencies and is involved with smart contracts, which are mostly computer protocols designed to facilitate, verify, or enforce the negotiation or execution of a contract digitally [25]. these contracts allow the realization of credible transactions without third parties and are stored under blockchain technology. due to this technology’s potential, many industries have been working in order to adapt and adopt it in different business processes, the main criteria to decide to use or not the blockchain will define whether or not the assistance of intermediaries along the process. there are different opportunities to apply blockchain technology in varied industries as is described next. in agriculture, blockchain applications include food safety through traceability of provenance, information system, agro-trade, finance, crop certification, and insurance [26]. the provenance of products and traceability is one the most important application of blockchain in order to track the life-cycle of a product, information that can be used to make an informed purchase. prior works have been done in this field, e.g., [27, 28]. several bank entities are working together to adopt blockchain technology, mostly for the settlement of transactions between financial entities; this will reduce the time in which a transaction is reflected from one entity to another. blockchain platforms will also likely take some role in replacing the systems that now manage the distribution of electricity. as a smarter grid is built, blockchains may aid in facilitating dynamic signaling between producers and consumers, especially as the line between those roles continues to blur [22]. identity management systems based on blockchain can facilitate the identification process in crucial situations like an emergency [22]. all the data about the medical records of a patient is shared among all practitioners, making these records trustworthy and immutable [22]. social programs are susceptible to being affected by unethical practices, due to its sense this contribution cortes et al., [29] presents a model based on blockchain technology with a reliable structure, in which the transactions are registered in blockchain and smart contracts take control over the verification of the processes’ fulfillment. industry 4.0 [30] with the internet of things technologies could provide the links between blockchain technologies and different applications as supply chains [31], even allowing to recover process information. also, there are many opportunities for implementing blockchain technologies for circular economy strategies as is described by vogel [32]. in this sense, blockchain technology could be used in reverse logistics to reduce uncertainty about the quality and condition of used products [33]. the applications of blockchain technology previously described having in common that they seek to provide credibility in the information. in this sense, they apply the principles of immutability, traceability, and transparency of the transactions carried out on the blockchain. these principles are used to solve problems in which there is a risk of losing, modifying, or even adulterating information. it is precisely these characteristics that this study wishes to add to the quality management system towards mitigating the risk of executing unethical actions and providing confidence to stakeholders. paradigmplus (2021) 2:2 22 rafael bettín-díaz et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 blockchain-enabled software architecture proposal the blockchain-based software architecture to support iso 9001: 2015 certification, is proposed to improve the current certification structure. as we have seen in section 2, currently there is a welldefined process to achieve this, involving third parties in the process, in terms of consulting and auditing. an information system architecture without the use of blockchain, such as those that currently exist in the market, may make it unnecessary to use a consulting company to generate pre-audit readiness; however, this same information system, using blockchain technology, can facilitate the audit process, due to the way in which the blockchain is constructed, the traceability of the transactions generated during the preparation of the certification process can be determined. in this case, the proposal of using blockchain to support a quality certification under a specific standard, i.e., the same software architecture we already know with an additional layer, see figure 1, must consider the use of specifics blockchain components as presented in table 2, with the purpose table 2: principal components of a blockchain-software solution to support iso certification. blockchain components description node application each internet-connected computer has to run a specific application to be a participant in the blockchain network. in a more technical look, each computer must be able to process application-specific messages to generate an update on the ledger[34]. shared ledger is a logic component; it is a data structure managed inside the node application. each of the participants of the network, once they have installed the node application, has access to the shared ledger of the ecosystem they have access to each participant able to run as many applications they have permitted to use, according to the specific rules, smart contract, and payment when applying [34]. consensus is the process by which a network, or node, guarantees the ordering of transactions and confirms its validity, in order to validate the block of chain[35]. smart contracts these are an important component of a blockchain, which helps to encode automatic validation for a transaction that before was specified on a written contract[36] alerts this are applications that work together in the blockchain to monitorthe smart contract results and produce events [37] participants in a permission blockchain network like the one suggested in this paper; participants will be everyone involves within the organization that needs to communicate with the blockchain. for a public blockchain like bitcoin, anyone with a virtual wallet1 will be a participant of the network. virtual machine it is a representation of a machine by a machine; in software development, a virtual machine assigns specific capabilities for the software (disk space, processor, and memory) that would help to improve performance[38]. in blockchain, as the last logic component, it lives in the node application, and some of these virtual machines are called wallets [34]. paradigmplus (2021) 2:2 a blockchain-based approach to support an iso 9001:2015 quality management system 23 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 1: layered architecture for a blockchain-enabled qms of being able to validate conditions and business rules that are present in the information that is being stored into the blockchain[39]. given the fact that an audit model must generate action plans regarding the findings, they must be parameterized a priori, so that the necessary alarms can be generated to comply with the standard. the main component of this architecture (see figure 1) is the business process management layer, which is used to communicate information among different audiences, in this case between the serfigure 2: communication between the qms and blockchain paradigmplus (2021) 2:2 24 rafael bettín-díaz et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 3: process that occurs inside the blockchain every time, prior to store a transaction. vices layer and operative system layer. as shown in figure 2 the communication between qms and blockchain will be through rest2 api3. services, which endpoints are provided by the blockchain and will be connected to the qms server creating the block of chains (see figure 1). for each transaction performed in the qms that triggers a call to the blockchain, will start a process (see figure 3) to verify the validity of the transaction (consensus), validate the outcome of the process through the smart contract (validation), which will establish whether the processes are inside the control limits or not; with this information, it will send a response to the qms server (alert) for the business process owner to take actions on the process, that, in the end, for this kind of certification will be a consider an opportunity for improvement and finally store the information on the blockchain (block 1). this software architecture is proposed to work under a permission blockchain, which means, the administrator, in conjunction with the organization, will decide who gets to participate and their roles. 3.1 proof of development a qms is used by businesses to control vital activities through a set of policies, procedures, and processes for meeting customers’ requirements. several types of qms have been developed based on iso 9000 standard, which requires all intern operations to be documented and employees to follow them. one of the major principles of qm is decision-making based on evidence. data gathered allows for inspecting the actual result of the process and then comparing it to the goal established in the quality objective in each process. based on the above, through a proof of concept (poc), we developed two smart contracts to simulate the monitoring of a task. this implementation was not carried out to see the overall functionality of the qms but focused on checking the traceability of one activity, specifically the creation, review, and approval of one document. we chose ethereum, an open-source blockchain platform with smart contracts functionality, for this implementation. it provides the object-oriented programming language solidity, allowing easy business logic implementation and producing a digital ledger of transaction records. finally, it was tested on the decentralized applications ethereum ecosystem by using metamask. those contracts in solidity contain persistent data in state variables, and functions can modify these variables. in this test, the contract creatortask creates the contract responsible4task. when the contract responsible4task is created, its constructor method, a function declared with the constructor 2representational state transfer 3application programming interface paradigmplus (2021) 2:2 a blockchain-based approach to support an iso 9001:2015 quality management system 25 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . keyword, is executed once. algorithm 1 models the responsibility concept through a contract; the ’owner’ variable is the type ’address’ that holds a 20-byte value corresponding to the last 20 bytes of the hash of a public key. the tasks and the owners (task responsible) are identified with addresses for this design. the addresses are pre-fixed ’0x’ since this represents the hexadecimal format (base 16 notation). it manages the data algorithm 1 smart contract: responsible4task . ▷ declaration of state variables: task ∶ t askcreator owner ∶ address rolen ame ∶ bytes32 idemployee ∶ uint create a task owner procedure constructor(roln ame′, idemployee′) ▷ constructor method of the smart contract owner ← msg.sender task ← t askcreator(msg.sender) rolen ame ← rolen ame′ end procedure idemployee ← newid obtain rol function getrolename(.) return rolen ame end function rename rol function setrolename(newrolename) if msg.sender = address(task) then rolen ame ← newrolen ame end if end function obtain employee id function getemployeeid(.) return idemployee end function modify employee id function setemployeeid(newid) if msg.sender = address(task) then idemployee ← newid end if end function transfer task responsability function transferresponsability(newowner) if msg.sender! = owner then return ▷ returns control to the calling function end if owner ← newowner end function paradigmplus (2021) 2:2 26 rafael bettín-díaz et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . related to the employee in charge of the task, so the function exists to transfer responsibility to a new owner when the employee cannot make the task. algorithm 2 models the task concept, and only when a task is created a ’responsible4task’ contract must be constructed then. this contract creates a responsibility contract from the source code (and the binary). it manages the data related to the task, what it is about, where the evidence is stored (records and documents), its state, and who the employee is related to it. figure 4 is composed of three images. the black box on the left is the development environment just before deploying the contract. above it, the upper rectangle in light blue color identifies the account ’0x737d..31f’ and the amount of ether cryptocurrency in it; and the bottom rectangle indicates the contract name and its state. on the right is the graphical interface of metamask that is connected to the account rp1 ’0x737d..31f’ that belongs to the rospten network. since the test uses the ethereum blockchain, the account needs eth gas, an amount of ethereum cryptocurrency, which means an amount in ether to pay the cost for users to interact with the network. miners in the ecosystem set the price of eth gas based on the network’s computational power they consume to verify a transaction and put on a layer of security to the network. at the bottom is the notification window that shows up algorithm 2 smart contract: taskcreator . ▷ declaration of state variables: typet ask ∈{creation, review, approval} description ∶ string source ∶ string ownert ask ∶ responsible4t ask create a task function createtask(typetask’, description’, source’, role, idemployee): responsible4task typet ask ← typet ask′ description ← description′ source ← source′ ownert ask ← newresponsible4t ask(role, idemployee) return ownert ask end function obtain the responsibility description function getresponsibility(.) return description end function obtain the employee id of task responsible function getresponsibility(.) return ownert ask.getemployeeid() end function update the role name of task responsible function updaterolename(newdescription) ownert ask.changerolen ame(newdescription) end function update the employee id of task responsible function updateemployeeid(newid) ownert ask.setemployeeid(newid) end function paradigmplus (2021) 2:2 a blockchain-based approach to support an iso 9001:2015 quality management system 27 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 4: smart contract deploying on web platform when the transaction is done and the contract is ready for execution. figure 5 shows the taskcreator and responsible4task contracts. the taskcreator exposes the function ’createtask’ after it executes and is included in a block whose hash is ’0xf39fd...’ (left side). in this test, the task is to create a document defined in the field type with the value ’0’, and the description field stores the detail of the task, which is a ’communication plan’. the source field relates to the absolute address where the document is stored, since database, repository, or local file system. figure 5: test of a task creation manager must create the communication plan document paradigmplus (2021) 2:2 28 rafael bettín-díaz et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . and the data of the task responsible are bound with the ’role’ and ’id_employee’ fields. the top right picture shows the function ’getresponsibility’ that returns the description of the task. and the bottom right picture is the contract ’responsible4task’. the function ’getowner’ returns the owner of the task address which is associated with account ’0xf39f...’ 4 discussion the proposed architecture is attempting to make iso certifications more transparent. due to the blockchain layer, certification information will be available for review and verification by anyone with permission. also, a business will have an integrated system that will ensure that all processes involved in certification are being monitored. this requires sending signs when needed; collecting information regarding quality inspections, materials, products, and more, these transactions are verified by smart contracts on the blockchain and then make available for audit[39]. in this sense, is mandatory to establish which transactions will be recorded on blockchain, to avoid unnecessary efforts or resource consumption in this way. document control refers to the policies and procedures that should be in place to ensure that there is organizational accountability for records and demonstrate the effective operation of the quality management system: it is fundamental to all companies’ success. from a regulatory sense, it’s their way of proving your documentation is following requirements for compliance. organizations must be able to prove that they are meeting their quality management goals. documentation must be accurate, objective, and current in this regard, and practice must stand up to the scrutiny that a properly executed external audit will demand. the proposed architecture could help these tasks, allowing the organizations to establish and control their documents and records as evidence of conformity to requirements to obtain an iso 9001 certification. about this, in the qms is suggested to identify what documents or information blockchain must record and by how many times keep it. the product/service quality hinges on the efficacy of the organization’s system to manage your process’s quality. it is essential to have the right foundation, infrastructure, and tools to oversee and maintain that quality. in this sense, the architecture proposed allows the automation of the audit process, which means early alert systems to make the right decisions and keep the system efficient and trustworthy. 5 conclusions in this work, a software architecture based on blockchain is proposed and explained to support an iso 9001: 2015 qms. the main objective of guaranteeing the immutability, traceability, and transparency of the quality assurance system is possible through the blockchain’s transparent consensus mechanism. this makes it possible to verify transactions’ validity and how information has been modified or created in the process. in this manner, it supports qms in the certification process. the workflows’ execution of the software architecture proposed allows the automation of the regulatory processes by reporting and monitoring required data and checking compliance and approval processes. the blockchain layer seeks to improve the iso 9001:2015 certification audit process by maintaining the traceability and immutability of the records stored in it. the proposed architecture is oriented to process and provides an intuitive and straightforward way to facilitate communication among the participants. all these allow the organizations to be in a constant improvement environment and have evidence to make crucial decisions. also, the system could deliver more quality products and services if the processes are being performed and improved systematically. finally, with the evolution of this kind of solution, companies could expand and build an ecosystem for working together and self-regulated with quality standards like iso 9001:2015, and represent paradigmplus (2021) 2:2 a blockchain-based approach to support an iso 9001:2015 quality management system 29 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . an opportunity to provide customer confidence. authors’ information – rafael bettín-díaz has a degree in systems engineering from universidad del sinú, montería, colombia, and a m.sc. in processes engineering from universidad ean, bogotá, colombia. – camilo mejía-moncayo has a degree and m.sc. in mechanical engineering both from universidad nacional de colombia, bogota, colombia. currently he is ph.d. candidate at école de technologie supérieure, montréal, canada. – alix e. rojas has a degree in systems engineering from escuela colombiana de ingeniería, and a m.sc. in systems engineering from universidad nacional de colombia, bogota, colombia. currently, she is an associate professor at the universidad ean, bogota, colombia. authors’ contributions – rafael bettín-díaz participated in writing the original draft, conceptualization, and software development. – camilo mejía-moncayo contributed to the conceptualization of the project, methodology, reviewing, and editing. – alix e. rojas: contributed to the conceptualization of the project, methodology, reviewing, and editing. competing interests the authors declare that they have no competing interests. references [1] iso, “the iso survey of management system standard certfications -2017explanatory note background,” tech. rep. august, iso, 2018. [2] j.-a. medina-merodio, c. de-pablos-heredero, l. jimenez-rodriguez, l. fernandez-sanz, r. robina-ramirez, and j. andres-jimenez, “a framework to support the process of measurement of customer’s satisfaction according to iso 9001,” ieee access, vol. 8, pp. 102554–102569, 2020. [3] i. betlloch-mas, r. ramón-sapena, c. abellán-garcía, and j. pascual-ramírez, “implantación y desarrollo de un sistema integrado de gestión de calidad según la norma iso 9001:2015 en un servicio de dermatología,” actas dermo-sifiliográficas, vol. 110, no. 2, pp. 92–101, 2019. [4] i. montiel, b. w. husted, and p. christmann, “using private management standard certification to reduce information asymmetries in corrupt environments,” sep 2012. [5] j. a. lipuma, s. l. newbert, and j. p. doh, “the effect of institutional quality on firm export performance in emerging economies: a contingency model of firm age and size,” small business economics, vol. 40, no. 4, pp. 817–841, 2013. [6] b. ullah and z. wei, “iso certification, corruption and firm performance: a cross-country study,” tech. rep., iso, 2013. paradigmplus (2021) 2:2 30 rafael bettín-díaz et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [7] d. p. kafetzopoulos, e. l. psomas, and k. d. gotzamani, “the impact of quality management systems on the performance of manufacturing firms,” international journal of quality & reliability management, vol. 32, no. 4, pp. 381–399, 2015. [8] r. prada ospina and p. c. ocampo, “quality in relation to models and management systems in the automotive sector. a bibliographic review,” revista espacios, vol. 39, no. 16, 2018. [9] s. johana, m. marín, j. luis, g. díaz, j. luis, and g. reyes, “sistema de gestión de calidad y certificación iso 9001:2008-limitantes y desafíos para las pymes quality management system and iso 9001:2008 certification-barriers and challenges to smes,” revista espacios, vol. 39, no. 09, 2018. [10] k. g. dastidar, “effects of corruption and incompetence in the quality monitoring process,” in oligopoly, auctions and market quality, economics, law, and institutions in asia pacific, ch. 0, pp. 75–112, springer, february 2017. [11] j. a. rodríguez-corzo, a. e. rojas, and c. mejía-moncayo, “methodological model based on gophish to face phishing vulnerabilities in sme,” 2018 icai workshops, icaiw 2018 joint proceedings of the workshop on data engineering and analytics, wdea 2018, workshop on smart sustainable cities, wssc 2018, workshop on intelligent transportation systems, wits 2018 and workshop on empirical experiences on software reuse, weesr 2018, 2018. [12] r. m. nivia, p. e. cortés, and a. e. rojas, “implementation phase methodology for the development of safe code in the information systems of the ministry of housing, city, and territory,” in computational science and its applications – iccsa 2018 (o. gervasi, b. murgante, s. misra, e. stankova, c. m. torre, a. m. a. rocha, d. taniar, b. o. apduhan, e. tarantino, and y. ryu, eds.), (cham), pp. 34–49, springer international publishing, 2018. [13] “iso 9001:2015(en), quality management systems.” [14] n. orviz martínez, a. blanco gonzález, and c. del castillo feito, “legitimacy and corporate reputation: study from iso management system standards,” revista venezolana de gerencia, vol. 25, no. 91, pp. 976–994, 2020. [15] international organization for standardization, “quality management principles,” 2015. [16] h. yin and p. j. schmeidler, “why do standardized iso 14001 environmental management systems lead to heterogeneous environmental outcomes?,” business strategy and the environment, vol. 18, no. 7, pp. 469–486, 2009. [17] l. m. fonseca, “from quality gurus and tqm to iso 9001:2015: a review of several quality paths,” international journal for quality research, vol. 9, no. 1, pp. 167–180, 2015. [18] american society for quality, “what is iso 9001:2015 – quality management systems?,” american society for quality, 2018. [19] the british assessment bureau, “iso 9001: the ultimate beginner’s guide,” 2016. [20] international organization for standardization, iso 9001:2015 quality management system requirements. norway: gnv gl, 2015. [21] satoshi nakamoto, “bitcoin: a peer-to-peer electronic cash system.” https://bitcoin.org/ bitcoin.pdf, 2008. [22] m. peck, “reinforcing the links of the blockchain,” 2017. paradigmplus (2021) 2:2 https://bitcoin.org/bitcoin.pdf https://bitcoin.org/bitcoin.pdf a blockchain-based approach to support an iso 9001:2015 quality management system 31 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [23] m. gates, blockchain: ultimate guide to understanding blockchain, bitcoin, cryptocurrencies, smart contracts and the future of money. north charleston, sc, usa: createspace independent publishing platform, 2017. [24] f. milani, l. garcía-bañuelos, and m. dumas, “blockchain and business process improvement,” 2016. [25] m. crosby, nachiappan, p. pattanayak, s. verma, and v. kalyanaraman, “blockchain technology: beyond bitcoin,” applied innovation review, vol. june, no. 2, pp. 6–19, 2016. [26] v. s. yadav and a. r. singh, “a systematic literature review of blockchain technology in agriculture,” in proceedings of the international conference on industrial engineering and operations management, pp. 973–981, 2019. [27] r. bettín-díaz, a. e. rojas, and c. mejía-moncayo, “methodological approach to the definition of a blockchain system for the food industry supply chain traceability,” in computational science and its applications – iccsa 2018 (o. gervasi, b. murgante, s. misra, e. stankova, c. m. torre, a. m. a. rocha, d. taniar, b. o. apduhan, e. tarantino, and y. ryu, eds.), (cham), pp. 19–33, springer international publishing, 2018. [28] q. ding, s. gao, j. zhu, and c. yuan, “permissioned blockchain-based double-layer framework for product traceability system,” ieee access, 2019. [29] c. cortés, a. guzmán, c. a. rincón-gonzález, c. torres-casas, and c. mejía-moncayo, “a proposal model based on blockchain technology to support traceability of colombian scholar feeding program (pae),” in applied informatics (h. florez, m. leon, j. m. diaz-nafria, and s. belli, eds.), (cham), pp. 245–256, springer international publishing, 2019. [30] b. g. choi, e. s. jeong, and s. w. kim, “multiple security certification system between blockchain based terminal and internet of things device: implication for open innovation,” journal of open innovation: technology, market, and complexity, vol. 5, no. 4, 2019. [31] m. teodorescu and e. korchagina, “applying blockchain in the modern supply chain management: its implication on open innovation,” journal of open innovation: technology, market, and complexity, vol. 7, no. 1, p. 80, 2021. [32] j. vogel, s. hagen, and o. thomas, “discovering blockchain for sustainable product-service systems to enhance the circular economy,” 14th international conference on wirtschaftsinformatik, pp. 1493–1507, feb 2019. [33] m. kouhizadeh, q. zhu, and j. sarkis, “blockchain and the circular economy: potential tensions and critical reflections from practice,” production planning and control, vol. 0, no. 0, pp. 1–17, 2019. [34] neocapita, “the logical components of blockchain – neocapita – medium,” 2017. [35] hyperledger.org, “introduction to hyperledger business blockchain design philosophy and consensus.” [36] s. mery and d. selman, “make your blockchain smart contracts smarter with business rules,” © copyright ibm corporation 2017, p. 21, 2017. [37] x. liu, w. wang, h. guo, a. v. barenji, z. li, and g. q. huang, “industrial blockchain based framework for product lifecycle management in industry 4.0,” robotics and computer-integrated manufacturing, vol. 63, p. 101897, 2020. paradigmplus (2021) 2:2 32 rafael bettín-díaz et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [38] j. smith and ravi nair, “the architecture of virtual machines,” computer, vol. 38, pp. 32–38, 5 2005. [39] c. paris, “quality professionals: embrace blockchain or perish,” 2018. paradigmplus (2021) 2:2 introduction literature review quality management systems and iso 9001:2015 blockchain blockchain-enabled software architecture proposal proof of development discussion conclusions vol. 4, no. 2, pp 1 17, 2023 doi: 10.55969/paradigmplus.v4n2a1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . a framework for robust attack detection and classification using rap-densenet temitope samson adekunle id 1, toheeb adetoyese adeleke id 2, olakunle sunday afolabi id 3, oluwaseyi omotayo alabi id 4,�, adekunle olugbenga ejidokun id 5, godwin nse ebong id 6, and temitope a. bamisaye7 1colorado state university, fort collins, united states temitope.adekunle@colostate.edu 2ladoke akintola university of technology, ogbomoso, nigeria taadeleke50@student.lautech.edu.ng 3university of abuja, abuja, nigeria afolabi.olakunle2019@uniabuja.edu.ng 4first technical university, ibadan, nigeria engrseyialabi@gmail.com 5dominion university, ibadan, nigeria. g.ejidokun@dominionuniversity.edu.ng 6university of salford, salford, united kingdom. g.n.ebong@edu.salford.ac.uk, 6national open university of nigeria, abuja, nigeria. bamisaye9999@gmail.com abstract network attacks must be effectively identified and categorized to guarantee strong security. however, current techniques frequently have trouble correctly identifying and categorizing new attack patterns. this study presents a novel framework for reliable attack detection and classification that makes use of the complementary strengths of rap music analysis methods and densenet convolutional neural networks. this study employs feature extraction based on the attention pyramid network (rapnet) framework that has been proposed to extract features from the input data, and pigeon in binary. afterward, feature selection based on optimization algorithm (bpoa) is performed. following the selection of the ideal characteristics, densenet201, the attacks in botiot, cicids2017, and other systems are categorized using deep learning as well as cicids2019 datasets. additionally, the conditional generic adversarial extra data samples are provided for minority classes using the convergent gap analysis network (cgan), so the imbalanced data issue should be addressed. in contrast to the recent intrusion. the outcomes show that the model is capable of precisely detecting and accurately categorizing dos and ddos attacks with rates of 98.63%, 98.68%, and bot-iot, cicids2017, and cicids2019 all scored 98.78%. keywords: attack detection ⋅ classification rap-densenet ⋅ deep learning received: 2 june 2023 ⋅ accepted: 25 july 2023 ⋅ published: 2 august 2023. this article is distributed under the terms of the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. https://doi.org/10.55969/paradigmplus.v4n2a1 https://orcid.org/0000-0003-4828-6039 https://orcid.org/0009-0005-0003-8716 https://orcid.org/0000-0003-0443-0991 https://orcid.org/0009-0005-0027-5930 https://orcid.org/0000-0002-0133-4761 https://orcid.org/0000-0002-9899-8838 2 temitope samson adekunle et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 introduction systems for computer networks have been put in place to grease device connections and carry out essential commercial operations. on the primary functions of reality’s connection systems, nonetheless, this has a lesser impact. crucial diligence like banking, healthcare institutions, and service providers are vulnerable to insecurity pitfalls because of their substantial and pivotal reliance on computer networks [1, 2, 3]. due to this dependence, maintaining ideal networks is necessary to maintain availability, effectiveness, and safety. a security breach can significantly impact network performance, leading to insecurity and eventual network incompatibility. also, cyberattacks may affect knockouts, issues in armament systems, and nonpublic information releases. they might beget the loss of incredibly sensitive and priceless data, similar to sanitarium lines, military records, etc. likewise, they can disable phone and computer networks, making data unapproachable or rendering systems unworkable [4, 5]. banking and government networks are particularly vulnerable because of the tremendous value of the data they contain. the hackers steal the information (especially other people’s banking details) and profit from that information. there have been many different kinds of attacks that have resulted in internet abnormalities. similar attacks have been more common over the past ten years, posing a severe threat to the stability of networks due to the revision of multitudinous services [6, 7]. denial of service (dos) and distributed denial of service (ddos) attacks rank among the most important. dos attacks can be divided into two groups.: service outages and service flooding. the most hazardous assaults are ddos attacks. the iot network has endured severe losses due to the ddos attack. thus, iot druggies accordingly paid great attention to the vulnerabilities. multitudinous bias or systems work together to attack a single target, making it challenging to detect and disable the attacking bias [8, 9, 10]. cyber bushwhackers constantly use a botnet to intrude with internet structure. ddos attacks are delicate to identify and help in real-time, yet this approach has enormous mileage because attacks can have significant goods. lately, deep literacy has attracted important interest in attack discovery due to its effective print birth and literacy capacities, specifically in settings with massive datasets. without contextual information, deep literacy ways ultimately capture significant characteristics from the input data using multitudinous layers [11, 12]. thus, in this paper, densenet 201 grounded deep literacy is enforced to perform multi-class brackets on dos and ddos attacks. to break the imbalanced data issue, a tentative generative adversarial network (cgan) is enforced to perform data addition. subsequently, the point birth and selection are performed using a refined attention aggregate network (rapnet) and double chump optimization algorithm (bpoa). eventually, the attacks are detected and classified using the densnet201 classifier. the following are this paper’s primary objectives: to fix the problem of unbalanced data and boost the effectiveness of the suggested model, tentative generative adversarial network (tentative gan) grounded data addition is employed, rather than depending on the traditional point birth system, a deep literacy approach grounded on rapnet is espoused to prize the essential attributes from the raw network business data. there are five sections in this study. the study’s preface is presented in section 1, and affiliated work is described in section 2. the exploration’s methodology is explained in section 3, and the study’s findings are bandied in section 4. finally, section 5 delivered our conclusion. 2 literature review dos and ddos assaults are severe trouble to numerous associations because of their tremendous capability to bring down vulnerable waiters in a short period. as a result, many exploration plans have focused on stopping dos and ddos attacks. a number of creative experimenters have suggested defenses against dos and ddos attacks. below is a concise explanation of some of them. to effectively recognize and classify ddos attacks, wei et.al., [13] incorporate two deep literacy-grounded paradigmplus (2023) 4:2 a framework for robust attack detection and classification using rap-densenet 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ways called bus encoder (ae)-multi-layer perceptron (mlp). to perform point birth without mortal backing, ae was enforced by the authors. using the uprooted features, colorful kinds of ddos attacks were classified by mlp network. to assess the effectiveness of the suggested approach, large ddos attack samples from the cicddos2019 dataset were used employed in terms of f1-score, recall, perfection, and delicacy criteria. shroff et.at., [14] created a generative inimical network (gan) grounded dependable sensor for relating cyber-attacks. in this system, two distinct gan-grounded models were enforced. the first creator produced benign cases that nearly recalled benign samples from the dataset and the alternate creator was able of producing ddos cases that nearly recalled those from the dataset. also, the author built a model based on deep neural networks (dnns) that can differentiate between benign exemplifications and ddos scenarios in the dataset with various similarity values. for classifying network business, a deep neural network (dnn) grounded ids was created by [15, 16, 17]. they enforced a four-subcaste network, and each subcaste contains 136 neurons. to dissect the effectiveness of the suggested approach, multitudinous trials were carried out with colorful hyperparameter combinations, and the results were compared with other shallow and deep ann models. they used cicids2017 and nsl-kdd datasets with standard performance criteria for this assessment. they also created and tested 36 indispensable dnn model combinations, each producing different issues. to identify unknown ddos attacks, shieh et.al., [18] created a system that employs reconstruction error and distributes retired subcaste features. the deep hierarchical reconstruction nets (dhrnet) structure was used in this exploration to redact it with a 1d connected neural network using spatial position constraint prototype loss function. an arbitrary grade descent approximation grounded oneclass svm (support vector machine) was enforced to identify the unidentified patterns in the ensuing stage. the performance of this approach was assessed using the cicids2017 friday open dataset. using colorful machine literacy and point selection algorithms, alduailij et al., [19] offered a ddos-attack discovery system. originally, the most material attributes from the cicddos2019 and cicids2017 datasets were named using the random forest (rf) feature importance and mutual information approaches. subsequently, the attack discovery was performed by weighted voting ensemble (wve), grade boosting (gb), knearest neighbor (knn), logistic retrogression (lr), and rf, and the performance was estimated using f1score, recall, perfection, and delicacy. 3 methodology approach this section presents envisioned deep literacy-grounded intrusion discovery system and its highposition armature to classify colorful types of dos and ddos attacks. first of all, figure 1 shows a suggested armature with five important stages. they are, pre-processing, data addition, point birth, point selection, and bracket. originally, the raw data is pre-processed in several ways to exclude unwanted data. the imbalanced data issue is also fixed using a tentative generative inimical network (cgan)grounded data addition fashion, which improves the quality of the classifier. subsequently, the point birth is conducted on the stoked data using refined attention aggregate network (rapnet) grounded deep literacy fashion. from the uprooted features, the binary pigeon optimization algorithm (bpoa) is enforced to gain the important aspects. eventually, densnet201 grounded the classifier examines these characteristics, and categorizes the cyberattacks. 3.1 pre-processing including a data pre-processing phase in training results in more reliable training and a more precise model. as a result, during this stage, undesirable characteristics like "flow packets/s" equal to "infinity" or "nan" are removed. the duplicate rows are then eliminated, including the following ones: fwd avg bytes/bulk, bwd avg bulk/rate, fwd avg bulk/rate, fwd avg packets/bulk, bwd avg packets/bulk, bwd psh flags, and bwd avg bytes/bulk. paradigmplus (2023) 4:2 4 temitope samson adekunle et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 1: structure of the system the main objective of the strategy here is the multi-class categorization of ddos attacks. this requires the use of encoding. this study uses one hot encoder (ohe) to achieve this. for each label, a new column is created and given a value of 1 or 0, depending on whether the record belongs to that category. the data are normalized using l2 normalization after the encoding procedure. the l2 standard was applied to each column. in other words, equation 1, where x stands for each instance of a record, defines the properties of the dataset. ∣∣x∣∣2 = ¿ á áà n ∑ i=1 ∣xi∣2 (1) generally speaking, normalizing the dataset records speeds up training considerably. this normalization resulted in a model that was more accurate because it handled a wide range of dataset attributes consistently. 3.2 data augmentation using cgan using the augmented data sets created in this section, which closely match the original data, by using conditional generative adversarial network (cgan) approach. the same training process used by gan is also used by cgan but with restrictions on the labeling of the generated samples. there are two sections in this framework. they are generator (gr) and discriminator (ds) networks. these two networks fight for information that is generated artificially. in equations 2 and 3, yr stands for the input of random variables from the gr network, and ds(x) stands for the likelihood that the ds network will successfully predict the initial data x. integrating and maximizing the aforementioned equations yields the discriminator’s total loss function. since the discriminator’s goal is to accurately discriminate between authentic and fake samples presented in equation 4 l(ds(x), 1)= log(ds(x)) (2) l(ds(gr(yr)), 0)= log(1 − ds((gr(yr))) (3) l(ds) = max[log(ds(x))+ log(1 − ds((gr(yr)))] (4) paradigmplus (2023) 4:2 a framework for robust attack detection and classification using rap-densenet 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . in contrast to the discriminator, the loss function of the generator is presented by equation 5 l(gr) = min[log(ds(x))+ log(1 − ds((gr(yr)))] (5) the value function v (gr, ds) is specified in equation 6 based on considering the entire dataset. min max v (gr, ds)= min max(ex∼pdata(x)[log(ds(x))]) +(eyr∼pyr(yr)[log(1 − ds((gr(yr)))]) (6) therefore, eyr∼pyr(yr) stands for the estimated return of all arbitrary inputs ex∼pdata(x) represents the estimated return of original data samples, pyr(yr) and stands for data dissemination from the generator and the actual data distribution is represented by pdata(x). through transposed convolutional layers, the input feature vectors are up-sampled by the generator network. a series of transposed convolutional layers are used with different numbers of channels, including 64, 128, 256, and 512. the blocks at each level correspond to the size of the feature vector input. this model also generates samples with a structure like the given data. 3.3 point birth using rapnet from the pre-processed data, the effective features are uprooted by the refined attention aggregate network (rapnet). encoder and decoder structures are the foundation of this network, containing five stages. the first three stages of the complication process use 1 × 1 complication layers, and the following two stages use an atrous complication with 3 × 3 complication layers. the relu subcaste is used among the two complication layers to produce the non-linear representation, which retrieves low-subcaste specific features. to up-sample the high position point maps in all residual blocks, the deconvolution fashion is employed. because to conduct the point emulsion operation, all point chart sizes must be the same. after that, the side connection is equipped with the convolutional block attention module (cbam) to acclimate the point maps subcaste by subcaste. it’s used to drop false discovery and increase point birth delicacy. the final residual block of the conv5 stage is subordinated to the aggregate pooling module’s operation in the decoding route to acquire environment data. this module employs a four-position aggregate with caddy sizes of 1 × 1, 2 × 2, 3 × 3, and 6 × 6 and employs the global normal pooling operation. subsequently, the point emulsion process is performed by concatenating the point charts of the garbling network’s corresponding layers with the decoding network. eventually, the combined point charts are p2, p3, p4, and p5. to produce the emulsion, point chart p2, which is composed of the point maps p3, p4, and p5, a thick connection is employed in the point aggregate structure. this system allows for the accession of multi-layer fused point maps with rich semantic and spatial data for multiscale structure birth. the point aggregate network has chosen consecution operation rather of element-wise addition for thick and side connections among up sampled point charts. when the final fused point chart p2 is up sampled, the point birth results are produced 3.4 the binary pigeon optimization algorithm for feature selection in this section, we describe how the bpoa grounded point selection fashion is employed to elect an ideal subset of features from the input data. this fashion contains three essential drivers. they’re a compass, chart, and corner. suckers induce a chart for home by interpreting the earth’s glamorous forces in the chart and compass. assume that, in ndimension hunt space, the ith chump of masses can be represented as si = (si, 1, si, 2... si, n). another n-dimensional vector vli = (vli, 1, vli, 2... vli, n) can be used to indicate a chump’s haste, which represents how the chump’s position changes. the preliminarily visited position of the chump is denoted by pli = (pli, 1, pli, 2... pli, n), and the paradigmplus (2023) 4:2 6 temitope samson adekunle et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . final optimum position of the chump is denoted by g = ( g1, g2... gn). equations 7 and 8 determine how all suckers fly. v li(te + 1)= v li(te)× e−rt + rand ×(sg − si(te)) (7) si(te + 1)= si(te)+ v li(te + 1) (8) here, v li and si, respectively, represent the pigeon’s current velocity and location at a given time. sg stands for the world’s best solution, while rand denotes any number between 0 and 1. the landmark operator assesses each pigeon based on the fitness ratings. equation 9, in which only 50% of the pigeons are taken into account to calculate the desired location of the center pigeon, is used in all iterations to update the number of pigeons. every other bird adjusts its terminal at the same moment. t np(te + 1)= t np(te) 2 (9) here, t np stands for the number of pigeons present at any given time. the target location is found using the equation 10. equation 11 is used to update the positions of the other pigeons based on this. sc(te + 1)= ∑si(te + 1)× f itness(si(te + 1)) t np ∑f itness(si(te + 1)) (10) sc(te + 1)= si(te)+ rand ×(sc(te + 1)× si(te)) (11) sc, in this case, stands for the position of the center pigeon. instead of using the traditional poa, the bpoa describes the search space as an n-dimensional boolean lattice, upgrading the solution in the issue space to a continuous numerical location. additionally, the solution is superior to a hypercube’s corner. in order to decide whether to choose or not, a provided variable and a boolean solution vector are also employed. 1 denotes the parameter selected to include the datasets in this state, while 0 denotes anything else. based on the aforementioned factors, a fitness function (fsfn) was developed to address this issue and achieve the following balance between the two goals (see equation 12) fsfn = α∆er(c)+ β ∣y s∣ ∣t f ∣ (12) here α ∈ [0, 1] denotes the error rate’s weight of classifications, the total amount of features included in the dataset is denoted by ∣t f ∣, the size of the subset selected by the method is denoted by ∣y s∣, the error rate of a classifier is represented by ∆er(c), and the reduction feature’s importance is denoted by β = 1−α. instead of the amount of chosen features, the classification performance allows a crucial weight. 4 experiments and results the effectiveness of the suggested intrusion detection model is examined in this section through a number of investigations, and the findings are reported. windows 10 64-bit with a core i7 processor running at 2.70 ghz and 16 gb of ram were the operating systems used for all trials. the suggested method is put into practice using python with tensorflow as the backend. the hyperparameters include an adam optimizer, a learning rate of 0.001, a relu activation function, a batch size of 32, a momentum of 0.9 and 50 epochs, and a dropout of 0.9. the dataset is divided into training and testing for 70% and 30%, respectively. paradigmplus (2023) 4:2 a framework for robust attack detection and classification using rap-densenet 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 dataset description 4.1.1 bot-iot dataset the data set was released by khraisat et al., [20]. it contains more than 72 million recordings and a range of synthetic and real-world events. while dos and ddos-type packets make up the majority of the data set, there are four different forms of assault. this set is unbalanced, much like the unswnb15 data set. 4.1.2 cicids2017 dataset the cicids-2017 intrusion detection dataset was recently created by the canadian institute of cybersecurity. the date, destination, source ip addresses, assaults, protocols, destination, and source ports are used to label the cicids-2017 dataset. there are elements of real, realistic internet traffic in it. this dataset was compiled over five days and contains 2,830,743 records and 80 network traffic features. the dataset consists of a csv file with regular and unauthorized traffic and eight traffic surveillance periods. the many categories in this dataset include ddos, dos, ssh, brute force, ftp, botnets, infiltration, heartbleed, and web attacks. 4.1.3 cicids2019 dataset this dataset includes a variety of ddos assaults that can be executed using the tcp and udp network protocols. this dataset classifies attacks using exploitationand reflection-based invasions. the dataset contains more than 80 flow attributes. the dataset was gathered over the course of two days for training and testing analysis. the dataset includes attacks using snmp, ldap, udp-lag, mssql, syn, netbios, ntp, dns, and webddos. 4.2 evaluation metrics in equations 13 to 16, f 1, rec, p re, and acc represent the f1-score, recall, precision, and accuracy, respectively. moreover, f ln denotes the false negatives, f lp denotes the false positives, trn denotes the true negatives, and trp denotes the true positives. acc = trp + trn numberof samples (13) p re = trp trp + f lp (14) rec = trp trp + f ln (15) f 1 = 2 × precision × recall precision + recall (16) 4.3 result and discussion in this part, the suggested framework is tested on the chosen datasets (bot-iot, cicids2017, and cicids2019) through many experiments, and the findings are compared to other techniques. paradigmplus (2023) 4:2 8 temitope samson adekunle et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.1 analyzing the performance of the bot-iot dataset table 1 shows the evaluation of bot-iot performance on a dataset. the table demonstrates how well our suggested technique performs across all classes. in the ddos and dos categories, the bot-iot dataset achieves 99.87% and 99.68% acc, respectively. in comparison to other courses, the performance for stealing and reconnaissance is a little worse. with our suggested method, only 99% (theft) and 98.67% (reconnaissance) acc are reached in those classes. ddos assaults and reconnaissance attacks behave similarly, which is represented in the current feature set. this action makes it harder to distinguish between the two attacks using the model. table 1: multi-class classification on bot-iot dataset techniques pre rec f1 acc normal 99.78 99.89 99.74 99.69 ddos 99.93 99.96 99.89 99.87 dos 99.79 99.86 99.81 99.68 theft 99.11 99.32 99.14 99 reconnaisance 98.69 99.05 98.91 98.67 table 1 is illustrated visually in figure 2. the f1 (99.89%), rec (99.96%), pre (99.93%), and acc (99.87%) values of the ddos class are greater than those of all other classes, as can be seen from the figure. overall, all forms of attack have produced favorable effects. however, because it resembled conventional data, the reconnaissance class was given the lowest mark. because a few occurrences in the dataset were incorrectly assigned to a different class, theft-exfiltration also obtained the lowest results. figure 2: multi-class classification on bot-iot dataset paradigmplus (2023) 4:2 a framework for robust attack detection and classification using rap-densenet 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . the findings of the proposed strategy are compared with those of the current intrusion detection methods tested on the bot-iot dataset after the multi-class classification, as shown in table 2. the table demonstrates that the pre, acc, f1, and rec of the suggested approach are more significant than those of other existing techniques, indicating that the proposed framework significantly lowers the false positives in the majority of classes. the performance and accuracy of the support vector machine (svm) may have been higher when compared to all other techniques. because it erroneously categorizes different types of theft assaults. additionally, a lot of attacks were mistaken for ordinary packets, proving svm’s inability to identify intrusions. table 2: comparison of the proposed approach on bot-iot dataset techniques pre rec f1 acc svm [26] 89.60 89.35 89.34 89.35 xgboost [27] 99.38 99.57 99.47 98.96 knn [28] 99.04 99.03 99.04 99.03 c4.5 [29] 92.00 proposed 99.46 99.61 99.49 99.38 the overall performance of xgboost is better than any other method when compared to it. but the acc of the knn (k-nearest neighbor) is higher (99.03%) than the xgboost (98.96%). due to the fact that it handles multi-class instances with ease and produces superior acc than svm. c4.5 performs more efficiently than svm as well. however, one statistic (acc) (see figure 3) alone is insufficient to fully express how well the technique classifies incursions. not to mention, when compared to other strategies, the outcomes of the suggested strategy obtained using the bot-iot dataset reveal that our technique delivers more useful findings. figure 3: accuracy comparison of the proposed approach on bot-iot dataset paradigmplus (2023) 4:2 10 temitope samson adekunle et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.2 performance evaluation on the cicids2017 dataset table 3 shows the suggested model’s performance in multi-class classification on the cicids2017 dataset according to acc, pre, rec, and f1. the performance of the suggested method is highest for "benign" traffic detection (detection acc of 99.97%) and lowest for "sql injection" traffic detection (detection acc of 97.98%). the classifier performs poorly because there are few "sql injection" data points in the entire dataset. additionally, the behavior pattern of a "bot" attack resembles that of normal network traffic, making it more difficult for the suggested approach to reliably identify the attacks. thus, average performance results. attacks like heartbleed and sql injection were more correctly predicted than brute-force attacks. table 3 is illustrated visually in figure 4. table 3: multi-class classification on the cicids2017 dataset techniques pre rec f1 acc benign 99.95 99.93 99.95 99.97 ddos 99.89 99.85 99.87 99.92 infiltration 99.59 99.57 99.58 99.63 portscan 99.87 99.85 99.86 99.92 bot attack 98.85 98.81 98.83 98.89 pataror-ftp 99.89 99.86 99.87 99.95 parator ssh 99.23 99.21 99.22 99.27 bruteforce 99.67 99.65 99.66 99.71 xss 99.17 99.14 99.16 99.33 sql injection 97.56 97.52 97.54 97.98 ddos goldeneye 98.92 98.89 98.91 98.96 ddos hulk 99.67 99.65 99.66 99.73 ddos slowhttptest 98.68 98.66 98.67 98.71 ddos-slowloris 98.83 98.8 98.82 98.86 heartbleed 98.08 97.99 98.04 98.12 ’brute force’ and ’ddos hulk’ both earn identical pre (99.67%), rec (99.65%), and f1 (99.67%) results in terms of other performance scores. it proved that, in this configuration, the suggested classifier still exhibits symmetric behavior with regard to traffic classifications. additionally, acc and rec rates are crucial for evaluating the performance of the classifier for each attack. the statistics show that a class with a low acc has a high number of false positives. it means that ’benign’ classes are erroneously labeled as attacks. a model with low recall may also overlook genuine intrusion. therefore, acc and rec values must be sufficiently high to guarantee that the model operates at its best. the proposed model, as shown in figure 4, gets higher values for all the parameters that define the effectiveness of the strategy for multi-class categorization. table 4 contrasts the findings of the proposed methodology with those of existing methods on the cicids2017 dataset to demonstrate the usefulness of the suggested strategy. comparing our proposed strategy to existing approaches, the performance has improved. recurrent neural networks (rnn) and deep neural networks (dnn) obtain comparable results for f1. adaboost performs below par when compared to all other methods for classifying ddos attacks. however, compared to all other methods, this methodology has a higher rec (100%) value. this means that there aren’t many false negatives produced by this method. the recurrent neural network (rnn) (98% acc) and 1d-cnn (98.96% acc) approaches, on the other hand, perform better than comparable methods and result in a small amount of missed classification mistakes. it is not, however, greater than the suggested strategy (99.26% acc). it suggests that the suggested method is better suited for classifying and identifying ddos attacks. paradigmplus (2023) 4:2 a framework for robust attack detection and classification using rap-densenet 11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 4: multi-class classification on the cicids2017 dataset table 4: comparison of the proposed approach on bot-iot dataset techniques pre rec f1 acc decision tree [30] 97.5 85 90 96.67 dnn [31] 96 1d-cnn [32] 98.96 adaboost [33] 81.83 100 90.01 81.83 rnn [34] 96 97 96 98 proposed 99.19 99.15 99.17 99.26 4.4 performance evaluation on the cicids2019 dataset table 5 displays the outcomes of the multi-class classification carried out using the suggested method on the cicids2019 dataset. the suggested method, as shown in the table, excels at multi-class categorization and produces the best outcomes for each attack type. for each class, acc rates are better than 98%. the benign, dns, and ntp classes’ respective acc rates are 98. 76%, 98.71%, and 98.58%, highlighting their outstanding achievement. the performance of classification in other assault categories also results in the most significant outcome. figure 5 shows a graphic representation of table 5. the proposed classifier did not perform as well on the "webddos" and "mssql" classes as it did on other class types. testing has revealed that "webddos" and "mssql" attributes share a lot of characteristics. in order to effectively classify the traffic data for these classes, the classifiers need more crucial properties. despite this, because our model uses the efficient bpoa algorithm to choose pertinent attributes, the suggested method’s detection acc for "webddos" and "mssql" is 98.58% and 98.12%, respectively. it will lead to an improvement in the classifier’s overall acc. paradigmplus (2023) 4:2 12 temitope samson adekunle et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 5: performance analysis of the proposed technique on cicids2019 dataset (multi-class) techniques pre rec f1 acc benign 98. 78 98. 97 98 .78 98. 76 dns 98. 72 98. 71 98. 72 98. 71 ldap 98. 37 98. 71 98. 50 98. 63 mssql 98. 23 98. 15 98. 13 98. 12 ntp 98. 62 98. 64 98. 63 98. 58 netbios 98. 63 98. 72 98. 61 98. 50 snmp 98. 37 98. 40 98. 34 98. 41 ssdp 98. 28 98. 43 98. 31 98. 40 udp 98. 42 98. 53 98. 45 98. 63 syn 98. 40 98. 67 98. 48 98. 56 udp-lag 98. 20 98. 49 98. 49 98. 62 webddos 98. 19 98. 61 98. 39 98. 58 figure 5: multi-class classification on the cicids2019 dataset table 6 and figure 6 compared to other established intrusion detection systems, evaluate the suggested method. methods that were already in use were utilized, including the radial basis function neural network (rbfnn), bayes point machine (bpm), explainable neural network (xnn), and convolutional neural network (cnn). according to the table, the proposed technique successfully uses the cicids2019 dataset to achieve 98.42% f1, 98.39% rec, 98.41% pre, and 98.29% acc. adparadigmplus (2023) 4:2 a framework for robust attack detection and classification using rap-densenet 13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 6: comparison of proposed strategy using the cicids2019 dataset techniques pre rec f1 acc bpm [35] 98. 34 96. 89 96. 92 96. 90 rbfnn [36] 98. 21 98. 31 98. 29 98. 24 xnn [37] 98. 10 98. 31 98. 23 98. 29 cnn [38] 98. 42 98. 35 98. 46 98. 38 proposed 98. 42 98. 39 98. 41 98. 39 figure 6: analysis of proposed strategy with existing techniques on the cicids2019 dataset ditionally, xnn and cnn perform better than bpm. however, all of the approaches produce better outcomes (over 98%) in terms of pre. 4.4.1 impact of feature selection approach a binary poa approach is used to enhance the recommended intrusion detection process by picking the most important elements from the obtained features. the bpoa-based technique offers more information while minimizing the number of features on the bot-iot, cicids2017, and cicids2019 datasets. the performance of the suggested strategy with and without feature selection is shown in table 7, which demonstrates that the suggested strategy performs better when an effective bpoabased feature selection approach is used. for the bot-iot, cicids2017, and cicids2019 datasets, respectively, it only achieves 98.63%, 98.76%, and 98.68% acc without the feature selection method. after the feature selection procedure, the classifier performs better with the best possible collection of features and produces the best results without it. paradigmplus (2023) 4:2 14 temitope samson adekunle et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 7: comparison of proposed strategy using the cicids2019 dataset with feature selection dataset pre rec f1 acc bot-iot 98. 36 98. 71 98. 64 98. 45 cicids2017 98. 29 98. 16 98. 53 98. 38 cicids2019 98. 40 98. 56 98. 41 98. 56 without feature selection bot-iot 98. 43 98. 62 98. 67 98. 63 cicids2017 98. 17 98. 42 98. 71 98. 76 cicids2019 98. 30 98. 65 98. 64 98. 68 5 conclusion the demand for using more precise and effective ids has grown more critical due to the quick increase in network traffic and the development of intrusions. therefore, a deep learning-based network intrusion detection is implemented in this research. the results showed how well the suggested strategy performed when it comes to identifying and classifying cybersecurity threats. different performance criteria, including delicacy, fscore, recall (perceptivity), and perfection (discovery rate) have been used in the evaluation process to dissect the utility of the suggested models on the three standard datasets. in discrepancy to former attack discovery ways, the proposed frame achieves superior results with 98.63, 98.76, and 98.68 delicacy for botiot, cicids2017, and cicids2019 datasets, independently. this outgrowth is attained by the bpoa-grounded point selection system, which improves the data quality. grounded on the findings of this study, it’s determined that the recommended model will help produce a successful intrusion discovery system with a high discovery rate. future work will number developing the suggested ids to fete other attack types. also, the recommended strategy can be altered and used in a significant security operation. authors’ information – temitope samson adekunle received the b.tech. degree in pure mathematics from the federal university of technology, minna, nigeria, the m.tech. degree in mathematics (fluid dynamics) from the federal university of technology, minna, nigeria and also msc degree in data science from the university of salford, manchester, united kingdom. he is a ph.d. student at the department of computer science, colorado state university, fort collins, united state. his major research focus is in the aspect of artificial intelligent and machine learning, fluid dynamics, mathematics modeling, data engineering, simulation and modeling, big data mining. – toheeb adetoyese adeleke received the b.tech. degree in computer engineering at ladoke akintola university of technology, ogbomoso, nigeria where he is currently a master student. his major research focus is in the aspect of artificial intelligent, data science and information security. – olakunle sunday afolabi obtained the b.tech. degree in computer engineering from ladoke akintola university of technology, ogbomoso, oyo state, nigeria, the m.sc. degree in information technology (it) from noun-national open university of nigeria, abuja, nigeria. he is a ph.d. student at the department of computer science, university of abuja, abuja, nigeria. his major research focus is in the aspect of artificial intelligent and machine learning, simulation and modeling, data mining, artificial neural networks, and information security. – oluwaseyi omotayo alabi received the b.eng. and m.eng. degree in mechanical engineering paradigmplus (2023) 4:2 a framework for robust attack detection and classification using rap-densenet 15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (thermo-fluid) from the federal university of technology, minna, nigeria, and the university of ilorin, nigeria, respectively. he is a ph.d. student at the department of mechanical engineering, university of ibadan, nigeria. he has been engaged in research and teaching for more than two years. his major research focus is in artificial intelligent and machine learning, energy, heat transfer, simulation and modeling, aerodynamic, and computational thermal fluids. – adekunle olugbenga ejidokun received his bachelor’s degree from ladoke akintola university of technology, nigeria, in 2010. he also obtained a master’s degree from obafemi awolowo university, ile-ife, nigeria, at the department of computer science in 2016 and is currently pursuing his ph.d. at the same university. he is currently a lecturer in the department of computer sciences at dominion university, ibadan. his research interests include information systems, data analytics, data science, and machine learning. – godwin nse ebong is an accomplished professional passionate about technology and innovation. he graduated from caritas university in nigeria and the university of salford, respectively, with a bachelor of science in engineering and a masters in data science, both in the united kingdom. he is an expert in cloud engineering, data engineering, and data analysis. – temitope a. bamisaye received the b.sc. degree in communication technology from the national open university of nigeria, abuja, nigeria, the m.sc. degree in information technology from the national open university of nigeria, abuja, nigeria. his major research focus is in the aspect of artificial intelligence and machine learning, data science, internet of things, data mining, and information security. authors’ contributions – temitope samson adekunle contributed with the conceptualization, validation and supervision. – toheeb adetoyese adeleke contributed with the conceptualization and methodology. – olakunle sunday afolabi contributed with the resources, software, project administration, and writhing of original draft. – oluwaseyi omotayo alabi contributed with the methodology, editing and review, visualization, and supervision. – adekunle olugbenga ejidokun contributed with the conceptualization, validation and supervision. – godwin nse ebong contributed with the conceptualization and methodology. – temitope a. bamisaye contributed with the resources, software, project administration, and writhing of original draft. competing interests the authors declare that they have no competing interests. funding no funding was received for this project. paradigmplus (2023) 4:2 16 temitope samson adekunle et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . references [1] h. aydın, z. orman, and m. a. aydın, “a long short-term memory (lstm)-based distributed denial of service (ddos) detection and defense system design in public cloud network environment,” computers & security, vol. 118, p. 102725, 2022. [2] o. d. okey, s. s. maidin, p. adasme, r. lopes rosa, m. saadi, d. carrillo melgarejo, and d. zegarra rodríguez, “boostedenml: efficient technique for detecting cyberattacks in iot systems using boosted ensemble machine learning,” sensors, vol. 22, no. 19, p. 7409, 2022. [3] s. a. ajagbe, k. a. amuda, m. a. oladipupo, f. a. oluwaseyi, and k. i. okesola, “multiclassification of alzheimer disease on magnetic resonance images (mri) using deep convolutional neural network (dcnn) approaches,” international journal of advanced computer research, vol. 11, no. 53, p. 51, 2021. [4] r. k. batchu and h. seetha, “a hybrid detection system for ddos attacks based on deep sparse autoencoder and light gradient boost machine,” journal of information & knowledge management, vol. 22, no. 01, p. 2250071, 2023. [5] j. li, h. zhang, z. liu, and y. liu, “network intrusion detection via tri-broad learning system based on spatial-temporal granularity,” the journal of supercomputing, vol. 79, no. 8, pp. 9180– 9205, 2023. [6] d. teixeira, s. malta, and p. pinto, “a vote-based architecture to generate classified datasets and improve performance of intrusion detection systems based on supervised learning,” future internet, vol. 14, no. 3, p. 72, 2022. [7] v. ravi, r. chaganti, and m. alazab, “recurrent deep learning-based feature fusion ensemble meta-classifier approach for intelligent network intrusion detection system,” computers and electrical engineering, vol. 102, p. 108156, 2022. [8] m. i. kareem and m. n. jasim, “fast and accurate classifying model for denial-of-service attacks by using machine learning,” bulletin of electrical engineering and informatics, vol. 11, no. 3, pp. 1742–1751, 2022. [9] a. a. alqarni, “majority vote-based ensemble approach for distributed denial of service attack detection in cloud computing,” journal of cyber security and mobility, pp. 265–278, 2022. [10] a. fatani, a. dahou, m. a. al-qaness, s. lu, and m. a. elaziz, “advanced feature extraction and selection approach using deep learning and aquila optimizer for iot intrusion detection system,” sensors, vol. 22, no. 1, p. 140, 2021. [11] m. v. gaur and r. kumar, “m-lstm: multiclass long short-term memory based approach for detection of ddos attacks,” mathematical statistician and engineering applications, vol. 71, no. 3s2, pp. 1375–1394, 2022. [12] j. halladay, d. cullen, n. briner, j. warren, k. fye, r. basnet, j. bergen, and t. doleck, “detection and characterization of ddos attacks using time-based features,” ieee access, vol. 10, pp. 49794– 49807, 2022. [13] y. wei, j. jang-jaccard, f. sabrina, a. singh, w. xu, and s. camtepe, “ae-mlp: a hybrid deep learning approach for ddos detection and classification,” ieee access, vol. 9, pp. 146810–146821, 2021. paradigmplus (2023) 4:2 a framework for robust attack detection and classification using rap-densenet 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [14] j. shroff, r. walambe, s. k. singh, and k. kotecha, “enhanced security against volumetric ddos attacks using adversarial machine learning,” wireless communications and mobile computing, vol. 2022, pp. 1–10, 2022. [15] h. azzaoui, a. z. e. boukhamla, d. arroyo, and a. bensayah, “developing new deep-learning model to enhance network intrusion classification,” evolving systems, vol. 13, no. 1, pp. 17–25, 2022. [16] s. a. ajagbe, o. a. oki, m. a. oladipupo, and a. nwanakwaugwum, “investigating the efficiency of deep learning models in bioinspired object detection,” in 2022 international conference on electrical, computer and energy technologies (icecet), pp. 1–6, ieee, 2022. [17] s. a. ajagbe and m. o. adigun, “deep learning techniques for detection and prediction of pandemic diseases: a systematic literature review,” multimedia tools and applications, pp. 1–35, 2023. [18] c.-s. shieh, t.-t. nguyen, c.-y. chen, and m.-f. horng, “detection of unknown ddos attack using reconstruct error and one-class svm featuring stochastic gradient descent,” mathematics, vol. 11, no. 1, p. 108, 2022. [19] m. alduailij, q. w. khan, m. tahir, m. sardaraz, m. alduailij, and f. malik, “machine-learningbased ddos attack detection using mutual information and random forest feature importance method,” symmetry, vol. 14, no. 6, p. 1095, 2022. [20] a. khraisat, i. gondal, p. vamplew, j. kamruzzaman, and a. alazab, “a novel ensemble of hybrid intrusion detection system for detecting internet of things attacks,” electronics, vol. 8, no. 11, p. 1210, 2019. paradigmplus (2023) 4:2 introduction literature review methodology approach pre-processing data augmentation using cgan point birth using rapnet the binary pigeon optimization algorithm for feature selection experiments and results dataset description bot-iot dataset cicids2017 dataset cicids2019 dataset evaluation metrics result and discussion analyzing the performance of the bot-iot dataset performance evaluation on the cicids2017 dataset performance evaluation on the cicids2019 dataset impact of feature selection approach conclusion vol. 3, no. 1, pp 1 18, 2022 doi: 10.55969/paradigmplus.v3n1a1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . multi-criteria and multi-expert requirement prioritization using fuzzy linguistic labels giovanni daián rottoli id 1,� and carlos casanova id 1 1grupo de investigación en inteligencia computacional e ingeniería de software (giicis), universidad tecnológica nacional facultad regional concepción del uruguay, concepción del uruguay, argentina. {rottolig, casanovac}@frcu.ut.edu.ar abstract requirement prioritization in software engineering is the activity that helps to select and order for the requirements to be implemented in each software development process iteration. thus, requirement prioritization assists the decision-making process during iteration management. this work presents a method for requirement prioritization that considers many experts’ opinions on multiple decision criteria provided using fuzzy linguistic labels, a tool that allows capturing the imprecision of each experts’ judgment. these opinions are then aggregated using the fuzzy aggregation operator mliowa considering different weights for each expert. then, an order for the requirements is given considering the aggregated opinions and different weights for each evaluated dimension or criteria. the method proposed in this work has been implemented and demonstrated using a synthetic dataset. a statistical evaluation of the results obtained using different t-norms was also carried out. keywords: requirement prioritization ⋅ fuzzy logic ⋅ linguistic labels ⋅ owa ⋅ software engineering received: 14 december 2021 ⋅ accepted: 5 february 2022 ⋅ published: 8 february 2022. 1 introduction according to ieee software engineering vocabulary [1], software engineering (se) is the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software; that is, the application of engineering to software. software is produced through a development process that aims to meet the needs of its users. such process is one of se’s main objects of study, for which several process models or lifecycles have been proposed, some of them more linear from the construction point of view, while others are iterative in nature [2]. in the latter, software construction occurs simultaneously with other activities such as design and planning. many iterative methodologies, such as unified process’, scrum or feature driven development fdd, propose to make a prioritization of the requirements. this task aims to achieve an ordering over the requirements according to their priority level. this prioritization helps when choosing what requirements will be implemented in each iteration, and it is useful to make decisions during iteration management. this article is distributed under the terms of the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. https://doi.org/10.55969/paradigmplus.v3n1a1 https://orcid.org/0000-0002-7623-2591 https://orcid.org/0000-0002-2142-2187 2 giovanni daián rottoli and carlos casanova . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . the criteria taken into consideration in requirement prioritization usually varies between methodologies. for instance, in scrum the prioritization is realized by the product owner by means of product backlog ordering [3]. although no attribute is specified in scrum guide, it is expected that the ordering is performed exclusively by value (i.e., the value or importance that the correspondent functionality can bring to the client), since the product owner represents the client in the development team. in others like the rational unified process (rup), both technical risk and business value are taken into account [4]. however, in practice, other criteria are taken into consideration when planning an iteration. besides business value and technical risk, reusability may be of particular interest as it allows effort to be saved for future functionalities. another criterion usually considered is the requirements’ capacity to be postponed if delays occurs in other tasks (a practice known as slack in extreme programming [5]). it seems reasonable then to introduce all attributes that may be relevant to sort the requirements. nevertheless, as the number of attributes considered in the process increases, comparisons become not only less significant, but also more difficult to perform. this phenomenon is produced by the curse of dimensionality [6], that is, more attributes introduce the impossibility of a-priori comparing requirements that are non-dominated (without using further preference information). on the other hand, multiple decision makers are usually involved in the prioritization process. in openup (open unified process), for instance, it is recommended that the whole team participate in the iteration planning [7]. again, in scrum, sprint planning is a ceremony in which the entire team participate and may also invite other people to attend in order to provide advice [3]. ultimately, the iteration planning is based on the expert judgement of the development team through an unstructured decision process where discussion and consensus among its members are key components. therefore, such a process has artisanal and heuristic characteristics that are far from the systematic, disciplined and quantifiable approach that defines se. for all this, it can be concluded that the prioritization process is typically multi-criteria and multi-person [8]. furthermore, an additional aspect that characterizes the prioritization process is that the valuations assigned to each requirement’s attributes suffer from the cone of uncertainty’s effects [9]. that is, such valuations are only estimates of the real values of the attributes, which will become more certain as time passes. this uncertainty comes from various sources and can take different forms, such as nonspecificity (several alternatives on which it has not yet been decided, for example, scope or design), imprecision (lack of definite or sharp distinctions between categories, for example, when estimating a feature’s complexity as low or moderate), strife (conflicting evidence or judgements, for instance, from multiple experts), among others [10]. in this sense, the use of estimates in the form of numbers or classic qualitative ordinal scales constitutes a methodologically questionable approach, since it implicitly assumes that such attributes are known with certainty. moreover, the use of probability theory can also be questioned, as only can cope with only one dimension of uncertainty (conflict) [11]. consequently, it is necessary to make use of a framework that takes into consideration the multiple dimensions of uncertainty present in the requirement prioritization process. fuzzy logic [12] offers a theoretical framework that allows to capture the imprecision of expert judgements through the use of linguistic labels, which represent fuzzy sets or numbers. these labels can be ordered by a nonreciprocal fuzzy preference relation [13], where a label si is preferable to another sj to a certain degree that varies between 0 (si is not preferable at all to sj) and 1 (si is at least as preferable as sj), admitting intermediate values. these labels can be used as values of the different attributes of the requirements and obtain orders from them. fuzzy logic also offers tools for decision-making in contexts with multiple objectives and multiple decision makers [10], so it is judged as a highly valuable tool in solving problems such as requirement prioritization. this work extends the proposal made in [14] and [15] and is structured as follows. section 2 presents the proposed method for approaching the prioritization process. therein, subsection 2.1 introduces the fuzzy linguistic labels, 2.2 describes the relevant aggregation operators used, and 2.3 specifies the prioritization algorithm that produces the requirements ordering from the imprecise evaluations of multiple experts across multiple dimensions or attributes. section 3 presents the case paradigmplus (2022) 3:1 multi-criteria and multi-expert requirement prioritization using fuzzy linguistic labels 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . of study used to test the approach and an experimental study about the impact of changing the t-norm in the prioritization process. section 4 contrasts this proposal with related works. finally, section 5 collects the most relevant conclusions and identifies improvements that will be addressed in future work. 2 proposed method 2.1 fuzzy linguistic labels this article presents a method that uses only ordinal information through a tuple of linguistic labels sorted by their index. let s be a finite and totally ordered set of labels used to indicate preferences with odd cardinality, where ∀si, sj ∈ s, si ≥ sj → i ≥ j. the semantic of each linguistic term si is given by the ordered structure. for example, s1 = (low, m iddle, high). among the benefits associated with this ordinal fuzzy linguistic approach, two of them are primal. first, its simplicity, since this technique works directly taking into consideration the order of the labels in s. secondly, the fact that linguistic assessments are approximations that are handled when it is impossible or unnecessary to obtain more accurate values [16]. thus, the experts can provide evaluations for each requirement on each dimension of interest using this tool without focusing in selecting a numeric value, resulting in an intuitive and effortless task. given e, a set of experts, p , a set of requirements to be prioritized, and d, a set of dimensions or objectives to be evaluated by the experts, the method must be provided with ∣e∣ matrices o∣p ∣×∣d∣, where each ok(l, m) ∈ s represents the opinion of expert ek ∈ e on how well the requirement pl ∈ p meets the dimension (criterion) dm ∈ d. additionally, each expert e ∈ e can be linked to an importance degree using also the ordinal scale s so their opinion will be ponderated accordingly. the same can be done with the evaluated dimensions, so each dimension d ∈ d weighs differently in the prioritization process. 2.2 aggregation in order to aggregate the ∣e∣ aforementioned experts’ opinions, an induced ordered weighted averaging operators (iowa) is used to generate a single general matrix with the combined opinions of all the experts. more specifically, in this work we use the majority guided linguistic iowa (mliowa) operator proposed in [16] because it solves common problems related to the semantic of aggregated linguistic values, and it is a majority guided operator, so the weights used in the aggregation process are induced considering the majority of values in the expert importance degree vector, allowing the multiple experts’ evaluations to be aggregated in a single one for each requirement in each dimension. nonetheless, it is important to note that any other iowa operator with similar characteristics can be used, for instance, based on another aggregation criterion than that of the majority. first, let n eg, m ax and m in be a unary operator and two binary operators respectively, defined on a linguistic set s ={s1, s2, . . . , s∣s∣} as presented in equations 1 to 3. n eg(si)= s∣s∣−i+1 (1) m ax(si, sj)= smax(i,j) (2) m in(si, sj)= smin(i,j) (3) paradigmplus (2022) 3:1 4 giovanni daián rottoli and carlos casanova . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . the mliowa operator is a function φiq ∶ (s × s) ∣e∣ → s, defined as follows, where i is the vector of importance related to the experts, and q is a linguistic fuzzy quantifier representing the concept of majority in the aggregation presented in equation 4. φiq((i1, p1), ⋯,(i∣e∣, p∣e∣))= sk (4) with sk ∈ s and k as presented in equation 5. k = round( ∣e∣ ∑ i=1 wi ⋅ ind(pσ(i))) (5) such as: 1○ ind ∶ s →{1, ..., ∣s∣} such that ind(si)= i 2○ σ ∶ {1, ..., ∣s∣}→{1, ..., ∣s∣} is a permutation such that uσ(i+1) ≥ uσ(i), ∀i = 1, ..., ∣e∣− 1. 3○ the order inducing values ui are calculated using the importance degrees ii as shown in equations 6 and 7. ui = supi + ind(ii) 2 (6) supi = ∣e∣ ∑ j=1 supij∣supij ={ 1 if ∣ind(ii)− ind(ij)∣< α ∈{1, ..., ∣s∣} 0 otherwise (7) 4○ and, lastly, the weighting vector w̄ calculated using the order inducing values as shown in equation 8. wi = q( uσ(i) ∣e∣ ) ∑∣e∣j=1 q( uσ(j) ∣e∣ ) (8) for further information about the mliowa operator, see the original publication [16]. 2.3 prioritization algorithm in a previous work, an algorithm for multi-criteria requirement prioritization was presented [14]. in this work, a variation of that work that uses only linguistic labels and includes multiple expert opinion is shown. given a set of linguistic labels s, a set of experts e, a set of requirements to be prioritized p , a set of dimensions or criteria d, ∣e∣ matrices o∣p ∣×∣d∣, just as was defined in section 2.1, and also given an expert importance degree vector i ∈ s∣e∣ and a criteria weight vector ρ ∈ s∣d∣, the steps of the algorithm are the following: first, the experts’ opinions must be aggregated using the mliowa operator as stated in section 2.2 using as input the ∣e∣matrices oi and the experts importance degree vector i, producing a matrix m ∣p ∣×∣d∣ with the aggregated experts’, where each mij ∈ s represents the aggregated opinion of the set of experts about requirement pi ∈ p in relation to the dimension dj ∈ d. the next step is to build a comparison matrix for each dimension using as input the matrix m. for a specific dimension dj ∈ d, the comparison matrix c dj represents the fuzzy preference relation paradigmplus (2022) 3:1 multi-criteria and multi-expert requirement prioritization using fuzzy linguistic labels 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . rdj ⊆ p × p , where µrdj , defined as shown in equation 9, is the degree to which the requirement pi ∈ p is at least as good as pk ∈ p according to the criterion dj. c dj ik = µrdj (pi, pk)= ⎧⎪⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎪⎩ 1 if ind(mij)≥ ind(mkj) 0 if ind(mij)< ind(mkj)− β 1 − ∣ind(mij)− ind(mkj)∣ β + 1 otherwise (9) with β ∈ n, 0 < β < ∣s∣. given a dimension j, when ind(mij) ≥ ind(mkj), i.e., when pi’s label is at a position greater or equal to the position of pk’s label, then pi is at least as preferable to pk in maximum degree regarding dimension j (e.g., at s1, middle is at least as good as low at grade 1). further, when ind(mij)< ind(mkj)− β, i.e., when the label of pi is at more than β positions of pk, then pi is not at all preferable to pk (e.g., at s1, with β = 1, low is at least as preferable as high at grade 0). lastly, when the position of pi’s label is at β or fewer positions than the position of pk’s label, then it is still possible that pi is preferable to pk, but in a lesser degree, proportional to the distance between positions (e.g., at s1, with β = 1, low is as preferable as middle at grade 0.5). such possibility exists due to the imprecision in the experts’ evaluations. hence, β is a parameter that reflects the overlapping between the fuzzy linguistic labels and depends on the characteristics of the addressed problem. then, these preference relations represented by each comparison matrix need to be weighted considering the criteria weight vector to obtain the pondered preference relations. to do this, an implication approach similar to the one presented by [17] can be used according to equation 10, where ↝ is the fuzzy implication symbol and � is the fuzzy t-conorm used to define it. this would be the fuzzy version of the logical equivalence p → q ≡ ¬p ∨ q µ ρj rdj (pi, pk)= ρj↝µrdj (pi, pk)=�(1 − ind(ρj) ∣s∣ , µrdj (pi, pk)) (10) then, the global fuzzy preference relation rg is the intersection rg = ∣d∣ ⋂ j=1 r ρj dj using the classic t-norm min as shown in equation 11. µrg(pi, pk)=⊺min dj∈d {µρj rdj (pi, pk)} (11) the global fuzzy preference relation is used then to build a new relation: the strict global fuzzy preference relation rs. the complement of µrs(pi, pk) (equation 12) represents the degree to which pi does not dominate pk, and can be used to calculate the degree to which an alternative pk is not strictly dominated by any other alternative. this last fuzzy set is called fuzzy non-dominance set and is calculated as shown in equation 13 [13]. µrs(pi, pk)= max{µrg(pi, pk)− µrg(pk, pi), 0} (12) µrn d(pk)= min pi {1 − µrs(pi, pk)}= 1 − max pi {µrs(pi, pk)} (13) the best alternative pn d ∈ a is the requirement with the highest membership degree to the fuzzy non-dominance set rn d. also, when this value is equal to 1, the decision is said to be non-dominated and non-fuzzy. borzęcka [18] proposes using the non-dominance fuzzy set to make the prioritization, but this may not be appropriate if there are alternatives that are not strictly dominated by others. for example, if there are 3 alternatives a, b and c, such that a dominates b, b dominates c and, transitively, a dominates paradigmplus (2022) 3:1 6 giovanni daián rottoli and carlos casanova . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . c, b and c would have their membership degree equal to 0 making them equally preferable according to [18], which is not true. in order to solve this problem, the most preferable solutions pn d can be removed from the set, then the membership function of the remaining alternatives to the strict global fuzzy preference relation can be calculated again and finally their non-dominance degree. this procedure is repeated until there is no alternative left to be prioritized. this method fits problems with either small or large numbers of requirements because it is possible to compose a single global fuzzy relation from an arbitrary number of dimensions. for this, fuzzy preference relations are used to evaluate multiple criteria with different degrees of importance in order to classify the alternatives considering the stakeholders’ opinions, weighted by their relevance. then, a decreasing order of the solutions is obtained taking into consideration all the aforementioned aspects. the structure of the algorithm is described in algorithm 1. algorithm 1 proposed prioritization algorithm require: s ▷ linquistic label set require: p ▷ requirement set require: d ▷ dimensions require: e ▷ experts require: ∣e∣× mi ∈ s∣p ∣×∣d∣ ▷ experts’ opinion matrices require: i ∈ s∣e∣ ▷ experts’ importance degree vector require: ρ ∈ s∣d∣ ▷ dimensions’ importance degreee vector ensure: partial order of requirements 1: m = aggregate experts’ matrices using mliowa operator 2: c g = initialize the global fuzzy preference relation matrix 3: for dj ∈ d do 4: c dj = build comparison matrix according to dj using m 5: c dj = ponderate dimension using ρj through the fuzzy implication formula 6: for i,k = {1,...,|p|} do 7: cg ik = ⊺min(mgik, c dj ik ) 8: end for 9: end for 10: order = [] 11: while p ≠ ∅ do 12: c s = build strict comparison matrix using c g 13: n d = build the non-dominance vector using c s 14: pn d = get the requirements with the highest non-dominance value 15: position = ∣order∣+ 1 16: orderposition = pn d 17: remove the requirements pn d from p and from c g 18: end while 19: return order 3 case study in order to illustrate how the proposed method works, a proof of concept using a generated dataset is shown below as a weak form of validation [19]. the dataset consists of five experts’ opinions about 10 requirements related to a content management system (cms) (see table 1) evaluated on 3 different dimensions: complexity degree of implementing the requirement, degree of reusability paradigmplus (2022) 3:1 multi-criteria and multi-expert requirement prioritization using fuzzy linguistic labels 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 1: requirement set used in the proof of concept. id requirement description 1 customize the user interface. 2 add content editor and content approver role. 3 provide a portal to the content provider with options. 4 email notifications to be sent to content provider. 5 conversion of the content to digital format. 6 quality check of the content. 7 check content format. 8 apply security over the content. 9 create package. 10 create poster for advertisements. of the implementation and importance to costumer. this set of dimensions gives better results in the requirement prioritization process according to [20]. the requirement set, the dimensions and the first expert evaluations were obtained from [14, 20] (see table 2). the rest of the opinions were generated arbitrarily. an implementation of this method and the data used for this experiment are publicly available, and the link to the repository can be found in the availability of data and material section of this article. firstly, in all the cases the fuzzy linguistic label set that was used is s, described below. in this case, the label “vl” means very low, “l” means low, “m” means medium, “h” stands for high and lastly “vh” stands for very high. however, it is important to note that the semantic of each label is given by its position in the tuple, as was mentioned in section 2.1. s = (s1 ∶ “vl”, s2 ∶ “l”, s3 ∶ “m”, s4 ∶ “h”, s5 ∶ “vh”) additionally, the expert importance degree vector i and the criteria weight vector ρ used were the ones described below: i =(i1 ∶ “m”, i2 ∶ “vh”, i3 ∶ “m”, i4 ∶ “m”, i5 ∶ “m”) ρ =(complexity ∶ “vh”, reusability ∶ “m”, importance ∶ “m”) using the aforementioned input, the first step of the algorithm aims to aggregate the experts’ opinions into a single general matrix. to do this, the mliowa operator is used. hereunder, the table 2: example of one of the experts’ opinions matrices used in the proof of concept. requirement complexity reusability importance 1 m l h 2 m vh vh 3 vh m h 4 vl vl vl 5 m h h 6 vh vh h 7 l l l 8 vl m vl 9 vl l l 10 l vl vh paradigmplus (2022) 3:1 8 giovanni daián rottoli and carlos casanova . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . calculations for the element m11 are presented as an example, corresponding to the requirement 1 and the criterion “complexity”, being the experts’ opinions for this pair the following: (e1 ∶ “m”, e2 ∶ “m”, e3 ∶ “l”, e4 ∶ “h”, e5 ∶ “m”) the support supi for each expert importance degree was calculated according to equation 7, considering α = 1, getting the vector sup = (4, 1, 4, 4, 4) as result. the first element of the vector was calculated with equation 14, and the rest of the elements were calculated equally. sup1 = 5 ∑ j=1 sup1j = 1 + 0 + 1 + 1 + 1 = 4 (14) the next step aims to get the order inducing value vector u, calculated with equation 15 according to equation 6. u =( 4 + 3 2 , 1 + 5 2 , 4 + 3 2 , 4 + 3 2 , 4 + 3 2 )= (3.5, 3, 3.5, 3.5, 3.5) (15) the vector u induces the order shown in table 3 and allows to get the weights to weigh the experts’ opinions using equation 8. in this case, the fuzzy quantifier “most of”, q, was defined by the parameters (0.3, 0.8). these weights can be seen in the same table mentioned above. table 3: induced order and weights σ expert u µq wi 1 e1 3.5 0.8 0.21 2 e3 3.5 0.8 0.21 3 e4 3.5 0.8 0.21 4 e5 3.5 0.8 0.21 5 e2 3 0.6 0.15 lastly, the experts’ were aggregated according to the induced order using the weights calculated before, using equation 16. k = round( 5 ∑ i=1 wi ⋅ ind(pσi)) (16) k = round(0.21 ⋅ ind(h)+ 0.21 ⋅ ind(m)+ 0.21 ⋅ ind(m)+ 0.21 ⋅ ind(l)+ 0.15 ⋅ ind(m))= = round(0.21 ⋅(4 + 3 + 3 + 2)+ 0.15 ⋅ 3)= round(2.97)= 3 then, the aggregated value for the requirement 1 evaluated on the criterion “complexity” is s3 ∶ “m”. the whole aggregated matrix is shown in table 4. after aggregating the experts’ opinions, the comparison matrices for each dimension were calculated using equation 9 (with β = 1)and then weighed using the equation 10 with the vector ρ. then, the global fuzzy preference relation was determined as the intersection of the three aforementioned comparison matrices. the result of this process can be seen as follows. paradigmplus (2022) 3:1 multi-criteria and multi-expert requirement prioritization using fuzzy linguistic labels 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 4: result of aggregating the five opinion’s matrices using the mliowa operator requirement complexity reusability importance 1 m l h 2 m vh vh 3 h m h 4 vl vl vl 5 m h h 6 vh vh h 7 l l l 8 vl m vl 9 vl l l 10 l vl vh c g = ccomplexity ∩ creusability ∩ cimportance = ⎛ ⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜ ⎝ 1 .4 .5 1 .4 0 1 .5 1 .5 1 1 .5 1 1 0 1 1 1 1 1 .4 1 1 .5 .4 1 1 1 .5 0 0 0 1 0 0 .5 .4 .5 .4 1 .5 .5 1 1 0 1 1 1 .5 1 .5 1 1 1 1 1 1 1 .5 .4 .4 0 1 .4 0 1 .5 1 .4 0 0 0 1 0 0 .5 1 .5 .4 0 0 0 1 0 0 .5 .5 1 .4 .5 .4 0 1 .4 0 .5 .4 .5 1 ⎞ ⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟ ⎠ lastly, the strict comparison matrix was computed, and then the partial order was generated by iteratively calculating the non-dominance vector and selecting those requirements with the biggest non-dominance membership value. the non-dominance vector calculated in each iteration as well as the non-dominated requirements are shown in table 5. the resulting order, then, is the one shown in the "selected" column, being the ones in the first row more preferable than the one in the second row and so on. if there are two or more requirements in a row, that means those requirements are equally preferable. table 5: algorithm iterations for requirements prioritization. an − shows the requirements that were selected previously. requirements i order 1 2 3 4 5 6 7 8 9 10 selected 1 1 0.0 0.5 0.4 0.0 0.0 1.0 0.0 0.0 0.0 0.4 6 2 2 0.4 1.0 0.9 0.0 0.5 0.0 0.0 0.0 0.4 2 3 3 0.4 1.0 0.0 1.0 0.0 0.0 0.0 0.5 3, 5 4 5 1.0 0.0 0.4 0.5 0.0 1.0 1, 10 5 7 0.4 1.0 1.0 0.5 7, 8 6 9 0.5 1.0 9 7 10 1.0 4 the results are consistent with the results obtained in [14]: when a single expert is provided, the two methods are equivalent. paradigmplus (2022) 3:1 10 giovanni daián rottoli and carlos casanova . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . c s = ⎛ ⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜ ⎝ 0 0 0 1 0 0 .6 .5 1 .0 .6 0 .1 1 .5 0 .6 1 1 .6 .5 0 0 1 0 0 1 1 1 .5 0 0 0 0 0 0 0 0 0 0 .6 0 0 1 0 0 .6 1 1 .1 1 .5 .6 1 1 0 1 1 1 .5 0 0 0 .5 0 0 0 0 .5 0 0 0 0 .6 0 0 0 0 0 0 0 0 0 .5 0 0 0 0 0 0 0 0 0 .6 0 0 .1 0 .1 0 ⎞ ⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟ ⎠ 3.1 t-norm comparison as mentioned in [15], the prioritization method proposed in this work uses the t-norm min to intersect the comparison matrices calculated for each evaluated dimension (algorithm 1, step 7). nonetheless, any other t-norm can be used. in order to understand if the selection of the t-norm used in this step produces a relevant impact in the result of the prioritization process, three different t-norms were compared by using random data matrices following the procedure specified in algorithm 2. this procedure uses input matrices generated randomly by taking values from the aforementioned vector s of linguistic labels. the number of requirements was constant and equal to 100 and the number of dimensions varied from 3 to 9, in order to understand whether the behavior associated with each t-norm depends on this parameter. with each configuration 500 comparisons were performed. it is also important to note that this randomly generated matrices simulate an aggregated matrix or the input matrix provided by just one expert, due to the lack of impact of this parameter in the objective of study. for the same reason, the dimensions were considered to have the same importance degree, so the weighting process was not performed. on the other hand, the t-norms considered in this work were the t-norm min (⊺m in), also called the gödel t-norm, for being the de-facto standard in the literature, the t-norm product (⊺p rod) for algorithm 2 procedure for t-norms comparison require: iterations ∈ n ▷ number of iterations require: nreqs ∈ n ▷ number of requirements require: ndims ∈ n ▷ number of dimensions require: t norms ▷ t-norms to be evaluated 1: t otalityset = ∅ 2: distances = ∅ 3: for i ∈ (1, 2, ..., iterations) do 4: d = randomm atrix(nreqs, ndims) 5: order = ∅ 6: for ⊺j ∈ t norms do 7: order(i, ⊺j)= p rioritize(d, ⊺j) 8: t otalityset(i, ⊺j)= evaluatet otality(d) 9: end for 10: for (⊺j , ⊺k) ∈ t norms × t norms do 11: distances(⊺j , ⊺k, i)= evaluatedistance(order(i, ⊺j), order(i, ⊺k) 12: end for 13: end for 14: return t otalityset, distances paradigmplus (2022) 3:1 multi-criteria and multi-expert requirement prioritization using fuzzy linguistic labels 11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . allowing the variables to interact with each other, and the łukasiewicz t-norm (⊺luk), for differ in the way the variables interact by using a summative formula. these t-norms are calculated respectively as shown in equations 17, 18 and 19. ⊺m in(a, b)= min(a, b) (17) ⊺p rod(a, b)= a ⋅ b (18) ⊺luk(a, b)= max(0, a + b − 1) (19) the procedure begins by generating in each iteration a random matrix of expert’s opinions using the parameters mentioned before. then, for each t-norm the prioritization process was carried on as specified in the previous sections. then, the totality degree of each generated order was calculated. this metric aims to get the degree to which an order is closer to be total as the ratio between the length of the longest chain (totally ordered set) that is subset of the evaluated order, and the length of its lineal extension, which is the same as the number of requirements. then, for each pair of t-norms, a distance was calculated between the orders generated by each one of them for a specific input matrix. this metric aims to find the average difference between the position of the requirements in both orders through the function shown in equation 20, where r is the set of requirements, and p os(x, o) is a function that returns the position of the requirement x in the order o. distance(o⊺j , o⊺k)= ∑ x∈r ∣p os(x, o⊺j)− p os(x, o⊺k)∣ ∣r∣ (20) as a result of the whole procedure, a set of totality values for each t-norm and a set of distance values between each pair of t-norms paired by input matrix was obtained. the distribution of these values can be seen in figure 1. using the wilcoxon signed-rank test [21], the non-parametric version of the paired t-test, it was tested if there are significative differences between the distribution of the totality variable for each considered t-norm. the results can be seen in table 6. according to the results, there is 88.30% of probability for the totality variable to be higher if the product t-norm is used instead of the min t-norm, 97.02% of probability to be higher if the product t-norm is used instead of the łukasiewicz t-norm, and a 87.75% to be higher if the min t-norm is used instead of the łukasiewicz t-norm, according to the common language effect size measure (cles) [22] and the rank-biserial correlation measure [23], as can be seen in the aforementioned table. on the other hand, as can be seen in figure 1, the totality degree decreases with the number of dimensions, specially using the min and the łukasiewicz t-norm. with the product t-norm, the totality value increases with low dimensionality but decreases after 5 dimensions. this behavior table 6: wilcoxon signed-rank test results hypothesis wilcoxon statistic p-value rbc cles h0 ∶ t otality(p rod)≤ t otality(m in) 2871928.0 0.0 0.999992 0.883074 ha ∶ t otality(p rod)> t otality(m in) h0 ∶ t otality(p rod)≤ t otality(luk) 2881200.0 0.0 1.0 0.970211 ha ∶ t otality(p rod)> t otality(luk) h0 ∶ t otality(m in)≤ t otality(luk) 2875044.0 0.0 0.997276 0.877522 ha ∶ t otality(m in)> t otality(luk) paradigmplus (2022) 3:1 12 giovanni daián rottoli and carlos casanova . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . (a) totality degree distribution (b) distances between t-norms figure 1: metrics evaluated in the t-norm comparison. 500 iterations with 100 requirements. could be produced due to the curse of dimensionality. if the number of dimensions is low, the extra dimensions could help the method to break ties, but if more dimensions are considered, the method could have problems in the comparison process. this hypothesis will be tested in future works. furthermore, the distance between paired solutions consistently increases for the pairs łukasiewicz and min, and łukasiewicz and product. thus, we could expect solutions to be different if łukasiewicz t-norm is used instead of any of the others. for the pair product t-norm and min t-norm, the solutions are more similar for higher dimensionality, having small totality degree. this also can be seen in figure 2, which shows how the probability of the totality value for the product t-norm to be higher than the value for the min t-norm decreases with the number of dimensions, so the orders tends to be more similar to each other, which also can be seen in the figure 1. additionally, the dispersion of the distance variable for this pair also increases, which does not occur for the other two pairs. in consequence, we can expect the orders obtained using the product t-norm to be more total than the ones obtained with the min and the łukasiewicz t-norms. in addition, the higher the dimensionality, the more partial these orders are. also, by using the łukasiewicz t-norm we cannot ensure paradigmplus (2022) 3:1 multi-criteria and multi-expert requirement prioritization using fuzzy linguistic labels 13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 2: cles value obtained through wilcoxon signed-rank test for each dimension. results similar to the ones obtained using the min or the product t-norm, but these last two t-norms tends to produce more similar results instead. these results can be used to choose the more appropriate t-norm according to both the characteristics of the problem and the characteristics of the decision-maker. more total orders, obtained using the product t-norm, may be preferred to the partial ones, for example, when trade-offs between objectives are allowed, while the opposite may be preferable when the decision-maker wants to break the ties according to their personal experience, specially when the number of criteria is low. this can be achieved, for example, by using the min t-norm. table 7, for example, shows different orders for the requirements used in the case study presented in section 3, each of which was obtained using a different t-norm and considering all the evaluated dimensions of equal importance degree. it can be noted that the t-norm min produced an order that is more partial than the one obtained using the product t-norm, being both orders consistent with each other. differences arise in pairs of requirements p1 and p10, and p7 and p8. in the last case, using the min t-norm, p7 and p8 are equally preferable (i.e., neither dominates the other), whereas with product and łukasiewicz t-norms, p7 has priority over p8. this is because product and łukasiewicz t-norms allow some interactivity between the criteria. particularly in this case, p7 is slightly superior to p8 in complexity (ccomplexity(p7, p8) = 1, ccomplexity(p8, p7) = 0.5) and importance (cimportance(p7, p8) = 1, cimportance(p8, p7) = 0.5), while p8 is slightly superior in reusability (creusability(p7, p8) = 0.5, creusability(p8, p7)= 1). aggregating the criteria using the min t-norm, cg(p7, p8)= min(1, 0.5, 1)= 0.5 and cg(p8, p7)= min(0.5, 1, 0.5)= 0.5, thus the strict relation values are cs(p7, p8)= cs(p8, p7)= 0, i.e., p7 and p8 do not dominate each other. however, it could be considered that, since p7 is slightly sutable 7: different orders for the requirement set used in the case study depending on the t-norm used in the prioritization process, considering the evaluated criteria to be equally important. position t-norm 1 2 3 4 5 6 7 8 9 10 totality min 6 2 3, 5 1, 10 7, 8 9 4 0.7 product 6 2 3, 5 1 10 7 8,9 4 0.8 łukasiewicz 6 2 3, 5 1 10 7 8,9 4 0.8 paradigmplus (2022) 3:1 14 giovanni daián rottoli and carlos casanova . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . perior to p8 in two of the three criteria, the former should have priority over the latter. this type of compensation can be taken into account if, for instance, the product t-norm is used. in such a case, aggregating the criteria yields cg(p7, p8) = 1 ⋅ 0.5 ⋅ 1 = 0.5 and cg(p8, p7) = 0.5 ⋅ 1 ⋅ 0.5 = 0.25, and thus cs(p7, p8) = 0.25 and cs(p8, p7) = 0, i.e. p7 is preferable to p8. similar results can be achieved with łukasiewicz’s t-norm. a similar analysis can be done for requirements p1 and p10 (p1 is slightly superior in complexity and reusability, while p10 is slightly superior in importance). lastly, all the tests performed in these evaluations are available in the availability of data and material section. 4 related work according to bukhsh et al [24], only a few articles uses fuzzy logic for requirement prioritization in software engineering. these studies differ from our proposal as describe hereunder. lima et al [25] suggested a framework that uses fuzzy linguistic terms parameterized using fuzzy numbers. in contrast, our methods do not need to use fuzzy numbers due to the fact that the semantics of each label is associated to its order in the fuzzy linguistic label set. additionally, our method allows integrating multiples decision makers or experts, which is not considered in the previously mentioned proposal. this also occurs in [26]. the authors designed a method for requirement prioritization easy to use and implement but do not consider multiple experts’. on the other hand, achimugu et al [27] considers the opinion of multiple stakeholders parameterized with triangular fuzzy numbers as in [25] but do not consider multiple objectives or criteria as our method does. then, franceschini et al [28] proposes a method for fusing multiples orders of priority, given by multiple experts with different degrees of importance. this method allows the stakeholders not to include into their order all the requirements, which is a tremendous advantage if the experts do not have the same degree of expertise on the different evaluated criteria. however, this method uses as input an order of requirements per expert, which is not the case of our algorithm that uses the experts’ opinions about the requirements instead. moreover, search-based approaches, like presented by tonella et al [29], propose to perform a prioritization incorporating order constraints, for instance, of technical precedence or business priority nature. however, the resolution method, based on an interactive genetic algorithm, does not help in the initial definition of such constraints, and our method could be used to that effect. furthermore, the proposal of [29] is based in the interaction with a single decision maker, while our method allows incorporating the opinion of multiple decision makers or experts. finally, the problem of grouping requirements and assigning them to software iterations or releases can be viewed as a prioritization. this problem has been formulated as an optimization problem and is known as the next release problem (nrp) or release planning (rp). there are many variants, both mono [30] and multi-objective [31]. however, as in the approach mentioned in the previous paragraph, information about the priority or value of requirements is often used as a measure of merit or quality to guide the search. similarly, the results of our approach can be used as parameters for the nrp and rp problems. to sum up, the algorithm proposed in this article allows the user to generate a partial order of a set of requirement based on multiple experts’ opinions with different importance degrees on multiple criteria, that also are weighed. moreover, the proposed method takes as input fuzzy evaluations using fuzzy linguistic labels, which allows the experts to provide their opinions in a more familiar way. the semantics of the linguistic labels is given by their position in the ordered fuzzy linguistic label set, so they do not depend on the definition of fuzzy numbers. the method is easy to implement and use. this can be seen in the implementation annexed to this article, making it highly applicable in the industry, which is a common limitation remarked in [24]. paradigmplus (2022) 3:1 multi-criteria and multi-expert requirement prioritization using fuzzy linguistic labels 15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 conclusion this work presents a method for requirement prioritization that uses the opinions of many experts on many decision criteria, expressed using fuzzy linguistic labels. the opinions are aggregated using a majority guided linguistic iowa considering weights for each expert, and then, the requirements are compared based on the aggregated experts’ opinions on the evaluated dimensions, which are also weighed by their importance. moreover, the weights linked to the criteria and to the experts are expressed using also fuzzy linguistic labels. the proposed method was demonstrated using a case of study that works as a initial form of validation. the algorithm was implemented in such a way that the user only has to provide the opinions and basic configurations to use it. this is a very desirable property. in future works, an empirical study has to be performed to complement the one presented in these pages, in order to understand how the proposal works in a real context with real users. in contrast to previous works, this article also presents an initial comparison of the different tnorms that can be used in the process, particularly in the intersection of each evaluated dimension. this study shows significant differences in the degree to which an order is more linear depending on the t-norm that was used to obtain it. nonetheless, the study was performed without considering different importance degree for each dimension. because of the use of a t-conorm in the weighting process that usually is associated with a t-norm, the results may vary if this extra factor is considered. this consideration remains for future works, as well as using different metrics to evaluate the obtained orders. finally, other improvements to be made in the future includes allowing the experts not to give an opinion for all the evaluated dimensions, considering consensus metrics in the prioritization process, using and comparing other iowa operators, and using a different linguistic label set for each dimension. acknowledgments the authors want to thank tomás casanova and kevin-mark bozell poudereux for proofreading the translation, and to the anonymous reviewers for their valuable comments and suggestions. authors’ information – giovanni daián rottoli is information systems engineer and ph.d. candidate in computer sciences. he is professor and researcher in the computational intelligence and software engineering research group from the universidad tecnológica nacional, facultad regional concepción del uruguay (argentina). – carlos casanova is ph.d. in engineering and information systems engineer. he is professor and main researcher in the computational intelligence and software engineering research group from the universidad tecnológica nacional, facultad regional concepción del uruguay (argentina). authors’ contributions – giovanni daián rottoli made the experiment design and execution and participated in the results analysis as well as writing the manuscript. – carlos casanova made the algorithm design and participated in the results analysis as well as writing the manuscript. paradigmplus (2022) 3:1 16 giovanni daián rottoli and carlos casanova . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . competing interests the authors declare that they have no competing interests. funding national technological university project siuticu0005297tc: “preference-based multi-objective optimization approaches applied to software engineering”. argentina. availability of data and material – gdrottoli/requirementprioritization: rp v1.1.0. 2021. doi: 10.5281/zenodo.5327643 – gdrottoli/requirementprioritizationexperiments_tnormscomparison: min-prod-luka comparison (v1.0). doi: https://doi.org/10.5281/zenodo.5771120 references [1] “iso/iec/ieee international standard systems and software engineering–vocabulary,” 2017. [2] p. bourque and r. e. fairley, eds., swebok: guide to the software engineering body of knowledge. ieee computer society, version 3.0 ed., 2014. [3] k. schwaber and j. sutherland, “the scrum guide,” scrum alliance, vol. 21, no. 19, p. 1, 2011. [4] p. kroll and p. kruchten, the rational unified process made easy: a practitioner’s guide to the rup: a practitioner’s guide to the rup. addison-wesley professional, 2003. [5] k. beck and c. andres, “extreme programming explained: embrace change. 2-nd edition,” 2004. [6] r. e. bellman and s. e. dreyfus, applied dynamic programming. princeton university press, 2015. [7] m. cossentino, v. hilaire, and v. seidita, “the openup process,” in handbook on agent-oriented design processes, pp. 491–566, springer, 2013. [8] t. luft, s. rupprecht, s. wartzack, et al., “a concept and prototype for a new app to support collaborative and multi-criteria decision making in product development,” in ds 87-1 proceedings of the 21st international conference on engineering design (iced 17) vol 1: resource sensitive design, design research applications and case studies, vancouver, canada, 21-25.08. 2017, pp. 389–398, 2017. [9] s. mcconnell, software project survival guide. pearson education, 1998. [10] g. j. klir and b. yuan, fuzzy sets and fuzzy logic: theory and applications. usa: prentice-hall, inc., 1994. [11] g. klir and m. wierman, uncertainty-based information: elements of generalized information theory, vol. 15. springer science & business media, 1999. [12] l. zadeh, “fuzzy sets,” information and control, vol. 8, no. 3, pp. 338–353, 1965. [13] s. a. orlovsky, “decision-making with a fuzzy preference relation,” fuzzy sets and systems, vol. 1, no. 3, pp. 155–167, 1978. [14] m. l. gabioud and c. casanova, “priorización en ingeniería de requerimientos con preferencias difusas,” in vii congreso nacional de ingeniería informática/sistemas de información, 2019. paradigmplus (2022) 3:1 10.5281/zenodo.5327643 https://doi.org/10.5281/zenodo.5771120 multi-criteria and multi-expert requirement prioritization using fuzzy linguistic labels 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [15] g. d. rottoli and c. casanova, “multi-criteria group requirement prioritization in software engineering using fuzzy linguistic labels,” in icaiw 2021: workshops at the fourth international conference on applied informatics 2021, ceur workshop proceedings, (new york, ny, usa), p. 16–28, association for computing machinery, 2021. [16] e. herrera-viedma, g. pasi, a. g. lopez-herrera, and c. porcel, “evaluating the information quality of web sites: a methodology based on fuzzy computing with words,” journal of the american society for information science and technology, vol. 57, no. 4, pp. 538–549, 2006. [17] r. r. yager, “a new methodology for ordinal multiobjective decisions based on fuzzy sets,” in readings in fuzzy sets for intelligent systems (d. dubois, h. prade, and r. r. yager, eds.), pp. 751– 756, morgan kaufmann, 1993. [18] h. borzęcka, “multi-criteria decision making using fuzzy preference relations,” operations research and decisions, vol. 22, 2012. [19] r. wieringa, design science methodology for information systems and software engineering. springer, 2014. [20] m. muqeem and m. r. beg, “a fuzzy based approach for early requirement prioritization,” international journal of computers & technology, vol. 15, no. 2, pp. 6480–6490, 2015. [21] f. wilcoxon, “individual comparisons by ranking methods,” in breakthroughs in statistics, pp. 196–202, springer, 1992. [22] k. o. mcgraw and s. p. wong, “a common language effect size statistic.,” psychological bulletin, vol. 111, no. 2, p. 361, 1992. [23] e. e. cureton, “rank-biserial correlation,” psychometrika, vol. 21, no. 3, pp. 287–290, 1956. [24] f. a. bukhsh, z. a. bukhsh, and m. daneva, “a systematic literature review on requirement prioritization techniques and their empirical evaluation,” computer standards & interfaces, vol. 69, p. 103389, 2020. [25] d. c. lima, f. freitas, g. campos, and j. souza, “a fuzzy approach to requirements prioritization,” in search based software engineering (m. b. cohen and m. ó cinnéide, eds.), (berlin, heidelberg), pp. 64–69, springer berlin heidelberg, 2011. [26] a. ejnioui, c. otero, and l. otero, “a simulation-based fuzzy multi-attribute decision making for prioritizing software requirements,” in proceedings of the 1st annual conference on research in information technology, riit ’12, (new york, ny, usa), p. 37–42, association for computing machinery, 2012. [27] p. achimugu, a. selamat, and r. ibrahim, “using the fuzzy multi-criteria decision making approach for software requirements prioritization,” jurnal teknologi, vol. 77, no. 13, 2015. [28] f. franceschini, d. maisano, and l. mastrogiacomo, “customer requirement prioritization on qfd: a new proposal based on the generalized yager’s algorithm,” research in engineering design, vol. 26, pp. 171–187, apr 2015. [29] p. tonella, a. susi, and f. palma, “interactive requirements prioritization using a genetic algorithm,” information and software technology, vol. 55, no. 1, pp. 173–187, 2013. special section: best papers from the 2nd international symposium on search based software engineering 2010. [30] a. bagnall, v. rayward-smith, and i. whittley, “the next release problem,” information and software technology, vol. 43, no. 14, pp. 883–890, 2001. paradigmplus (2022) 3:1 18 giovanni daián rottoli and carlos casanova . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [31] y. zhang, m. harman, and s. a. mansouri, “the multi-objective next release problem,” in proceedings of the 9th annual conference on genetic and evolutionary computation, gecco ’07, (new york, ny, usa), p. 1129–1137, association for computing machinery, 2007. paradigmplus (2022) 3:1 introduction proposed method fuzzy linguistic labels aggregation prioritization algorithm case study t-norm comparison related work conclusion vol. 4, no. 1, pp 1 17, 2023 doi: 10.55969/paradigmplus.v4n1a1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . continuous eye disease severity evaluation system using siamese neural networks muyideen abdulraheem1, idowu d. oladipo1, sunday adeola ajagbe id 2,�, ghaniyyat b. balogun1, mulikat bola akanbi1, and nissi o. emma-adamah3 1university of ilorin, ilorin, nigeria {muyideen,odidowu,balogun.gb}@unilorin.edu.ng,nissiea76@gmail.com 2first technical university, ibadan, nigeria sunday.ajagbe@tech-u.edu.ng 3kwara state polytechnic, ilorin, nigeria akanbiforu@gmail.com abstract evaluating the severity of eye diseases using medical images is a very essential and routine task performed in medical diagnosis and treatment. current grading systems which are largely based on discrete classification are unreliable and do reflect not the entire spectrum of eye disease severity. the unreliability of discrete classification systems for eye diseases is clear, as classification is subjective and done based on the personal opinion of various medical experts, which may vary. in a bid to solve these issues, this study proposes a system for determining the severity of eye diseases on a continuous range using a twin-convoluted neural network approach known as siamese neural networks. this system is demonstrated in the domain of diabetic retinopathy. samples of retinal fundus images from an eye clinic in india are taken as test cases to evaluate the performance of a siamese triplet network which attempts to find the distance between their image embedding. the outputs of the siamese network when a reference image is juxtaposed with a collection of images with distant severity categories (negative images), as well as when two reference images are compared to each other, are found to have a positive correlation (95%) with originally assigned severity classes. hence, these outputs indicate a continuous range of the severity and change in eye diseases. keywords: eye diseases ⋅ siamese ⋅ neural networks ⋅ convoluted neural network ⋅ classification received: 23 february 2023 ⋅ accepted: 15 march 2023 ⋅ published: 30 march 2023. 1 introduction eye diseases, if left untreated, can result in partial or complete loss of eyesight. early identification of many eye illnesses can help avoid visual loss. the assessment of the gravity of eye diseases and the change over time in medical images is a common and crucial activity. in diabetic retinopathy, for example, as a result of retinal blood vessel injury, vascular alterations might be a key indicator of disease progression and responsiveness to treatment. the study examines three eye diseases that threaten eye vision, diabetic retinopathy, cataracts, and diabetic macular edema. diabetes retinopathy (dr) is one of the most common eye disease impediments arising from diabetes and it is primarily this article is distributed under the terms of the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. https://doi.org/10.55969/paradigmplus.v4n1a1 https://orcid.org/0000-0002-7010-5540 2 muyideen abdulraheem et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . responsible for blindness in many patients with diabetes. it is caused by injury to the retinal blood vessels. the retina is responsible for the detection of light and transmission of a signal to the brain. these impulses are decoded by the brain for it to see the objects around it. early identification of retinal abnormalities can prevent diabetes, in addition to being critical in avoiding the advancement of dr diabetic retinopathy, as well as other problems that can result in visual loss. systems built on the identification of particular abnormal patterns or lesions are now used to automatically screen diabetic retinopathy using fundus imaging [1] for instance, microaneurysms (tiny saccular capillary dilations that look like dark red circular dots with sharp edges on the retina backdrop), exudates (which are lipid and protein deposits in the retina that cause brilliant yellowish-white lesions with conspicuous and uneven borders.), cotton wool spots (fluffy white spots on the retina arising from axoplasmic material accumulations inside the nerve fiber layer) and hemorrhages (a large amount of blood has accumulated in the retina.) [2, 3]. figure 1 shows the retinal fundus image of an eye while figure 2 illustrates the retinal fundus image cross-section. using medical images to assess the severity of eye diseases and their course over time is a task routinely carried out by medical practitioners [4]. for a variety of eye diseases, the process of the disease may have a wide range of unending severity, subject to modification with time. however, the severity of these diseases is often divided into ordinal categories (for example normal, moderate, and severe), and changes within these categories may not generally be understood. this is further complicated by the wide range of domain specialists’ interpretations of classes, which might lead to modifications in clinical therapy [5, 6]. as a result, rating severity and assessing change continuously has the potential to help better monitor disease progression, improve patient treatment, and improve medical research. deep learning algorithms are gaining a lot of traction in the medical imaging sector these days as a way to tackle a variety of issues. in radiology, an example is detecting diseases or anomalies in x-ray pictures and classifying them into various kinds or severity levels [7, 8]. the majority of published work has been concentrated on predicting discrete labels including separating patient images into distinct categories. however, in the course of assessing disease severity and change by predicting absolute class labels and comparing them, granularity past the absolute class labels is not attainable [6]. siamese neural networks, created to check the legitimacy of card (mainly credit card) signatures, are one method for analyzing the similarity between two images [9]. these networks consist of twin neural networks each able to learn an input vector’s hidden representation. in addition, the networks share the same configuration, parameters, and weights and the output may be regarded as the semantic resemblance between the extrapolated representations of both input vectors [10]. the euclidean distance of the networks’ last two layers may be determined, which measures the distance between the figure 1: retinal fundus image paradigmplus (2023) 4:1 continuous eye disease severity evaluation 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 2: retinal fundus image cross-section two images in terms of the imaging characteristics being trained on, like illness indicators. therefore, the task of evaluating the gravity of eye diseases on a continuous scale using images may be recast as a problem centered on metric learning. disease severity may therefore be reduced to the median of the euclidean distances obtained when an image is compared pairwise to a set of "normal" pictures [4]. however, this study proposes comparing an image-of-interest to both images which show the presence of the disease as well as “normal” images to better capture similarity and dissimilarity, crucial factors to consider when attempting to determine disease severity. the routine task of evaluating eye disease severity and change over time is currently done using grading systems that employ discrete severity categories. however, these grading systems are unreliable as they are subject to domain experts’ varying interpretations and opinions on severity category criteria for the diseases. the underlying continuous spectrum of severity is likewise not reflected by these distinct classifications. in addition, since patient visits are not always regular or guaranteed, the availability of multiple images per patient taken at different time points is unlikely, this poses a challenge to evaluating the severity of eye diseases on a continuous scale as well as predicting symptom trajectories from data taken at single time points. this study realizes these challenges and proposes a system that generates a quantitative severity score that adequately represents the underlying continuous nature of eye disease severity. this work aims to propose a system a continuous eye disease severity evaluation system using siamese neural networks. the following are the objectives of the study: – apply a sequence of image pre-processing techniques to retinal fundus images. – train a u-net for retinal vessel segmentation. – utilize triplet mining for selecting ideal triplet images for training – design a triplet siamese neural network for learning image embeddings. – evaluate results by calculating the correlation between the final output of the siamese neural networks and disease severity grade. 2 related work in a bid to develop efficient systems for evaluating and predicting eye disease severity, a plethora of studies have been conducted and a lot of varying techniques researched in multiple domains. ting paradigmplus (2023) 4:1 4 muyideen abdulraheem et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . et al., [11] reviewed articles centered on eye conditions like glaucoma and diabetic retinopathy, and macular diseases associated with age. in their study, they compiled a list of publications published between 2016 and 2018 and summarized them. they compiled a list of studies that employed fundus and optical coherence tomography pictures, as well as tl methodologies in their research. mookiah et al., [12] reviewed computer-aided diabetic retinopathy systems which are largely based on lesions and retinal vascular alterations. their review discusses the advantages of automated dr detection systems over manual ones. previous works were centered around feature extraction and image processing methods for detecting dr using a variety of computer-aided systems. acharya et al., [13] combined morphologybased image processing techniques and support vector machine techniques for automated dr detection. training the model in this study was made based on the presence or absence of four salient features, blood vessels, microaneurysms, exudates, and hemorrhages observed from fundus images. the study’s results present the use of a support vector machine (svm classifier which indicated an accuracy of 85 percent in predicting the unknown dr class with all four features proving significant. similarly, anant et al., [14] offered a strategy for detecting texture and wavelet characteristics using image processing and data mining. the results were produced for images from the diaretdb1 standard database and assessed using sensitivity, specificity, and accuracy metrics. the method yielded a model achieving an accuracy of 97.75%. colomer et al., [2] proposed a method for dr detection based on the combination of morphological (granulometry) and texture descriptors from retinal images. their suggested method eliminates the need for the segmentation of eye lesions prior to the categorization stage. their proposed methodology creates the basis for the exploration of higher-level computer-aided diagnostic algorithms for automated dr detection and classification. similarly, mushtaq et al., [15] apply a densely connected convolutional network for the early detection of dr. the proposed methodology provides a robust system for automated dr detection accomplished through a series of steps including data collection, preprocessing, augmentation, and modeling. the study employs preprocessing techniques such as data augmentation to address the class imbalance in the dataset, thereby outperforming a number of machine learning classifiers such as decision tree (dt), k-nearest neighbor (knn), support vector machine (svm) [16]. most of the research done regarding dr severity evaluation and prediction has been based on distinct dr severity classes such as non-proliferative, severe, moderate, mild, and normal. leeza et al., [17] proposed an enhanced automated technique for evaluating the gravity of diabetic retinopathy that does need not any preor post-processing stages. pathological explicit picture representation is incorporated into a learning outline with this method. areas of interest are identified to compute images’ descriptive features of the retina using a speeded-up robust features technique and a histogram of directed gradients to generate the dictionary of visual characteristics. the five severity levels of diabetic retinopathy include normal, mild, moderate, and severe non-proliferative diabetic retinopathy, as well as proliferative diabetic retinopathy, as determined by a radial basis kernel support vector machine and a neural network. in comparison to the stated state-of-the-art approaches, the suggested system achieved better results with 95.92 percent sensitivity and 98.90 percent specificity. a mechanism for classifying colored fundus images at various stages of non-proliferative diabetic retinopathy (npdr) was suggested by akram et al., [18]. the proposed system consists of a new hybrid classifier for retinal lesion detection, preprocessing, extraction of patient lesions, formulation of feature sets, and classification. preprocessing involves removing background pixels and isolating the blood vessels and optic discs from the retina images. this study combined a gaussian mixture model-based expansion of the m-mediods-based modeling technique to form an ensemble classifier targeted at achieving higher classification accuracy. prakash [19] proposed a severity analysis method for retinal images showing the presence of dr, involving several phases; pre-processing phase, segmentation phase, feature extraction phase, and classification phase. in the proposed method, after pre-processing and before analyzing the retinal images, the images are classified into normal and abnormal images using a neural network classifier based on paradigmplus (2023) 4:1 continuous eye disease severity evaluation 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . the mean, variance, entropy, area, diameter, and number of areas obtained from the segmentation optic disk of the retinal images, as well as the mean, variance, entropy, area, diameter, and number of regions acquired from the segmented blood vessels of the images. svm classifier is then used to evaluate dr severity based on the area and intensity of strong exudates and hemorrhages present in the images. the results proved that this work outperformed existing works in accuracy, with an average of 98% accuracy for both optic disks segmentation accuracy and blood vessels segmentation. the categorization of normal and pathological retinal images using neural networks achieved a 90% accuracy. different from the previous works, some more recent works sought to derive a quantitative measurement of severity in diseases of the retina (such as prematurity retinopathy and diabetic retinopathy) that can be used to keep track of disease progression during and subsequent treatment. gupta et al., [20] showed that a measurable prematurity retinopathy severity score generated by the system of deep learning may be used to follow illness progression in a group of newborns with disorders that required treatment. the study suggested that a measurable score of rop vascular severity produced automatically using deep learning may be applied to track disease retrogression in subsequent treatments. the main findings of this work were: the progression of rop’s measurable score of vascular severity over time was linked to the evolution of clinically observable disease (prior to treatment) and decline (upon treatment), newborns treated with bevacizumab showed a higher severity score at the start of the treatment and experienced faster disease decline compared to those who are laser photocoagulation-treated, patients who required re-treatment after laser treatment had a higher score of rop severity at the time of starting therapy and a faster rate of disease than those who were not. the technology proposed has the potential to track atypical and conventional disease progression patterns, detect the recurrence of illness, and provide an objective parameter to guide future therapy and post-treatment outcome. these findings might be applied to other medical and ophthalmologic conditions where treatment is presently based on subjective clinical criteria. to address the issue of subjectivity involved in clinicians performing the routine task of evaluating disease severity, li et al., [4] created a siamese neural network used to assess a single instance of disease severity and measure transformation between repeated patient visits to the clinician on a regular basis. the study demonstrated this approach in two domains; retinal images of retinopathy of prematurity (rop) and knee radiographs of osteoarthritis. the images used were evaluated and annotated with severity classes by expert clinicians. the researchers discovered that when compared to a group of unaffected (no presence of disease) reference images, the output of the siamese neural network for an image correlates with the disease severity score (0.87 for rop and 0.89 for osteoarthritis), both within and between clinical grading categories. in addition, using the siamese neural network, paired images from the same patient at two different times were juxtaposed, producing an additional continuous way of assessing change. importantly, the method used in the study does not necessitate manually pinpointing the pathology and merely requires two alternative labels for training (same against different). the study however, in training the siamese network, failed to explore other loss functions like the margin ranked loss and triplet loss functions which could potentially incorporate more information in the training labels, thereby improving the performance of the model. 3 methodology in this section, the study describes the approach taken to assess the severity of eye diseases on a continuous scale for fundus retinal images captured both at single instances of time and for the same patient longitudinally. paradigmplus (2023) 4:1 6 muyideen abdulraheem et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 data collection two datasets were used in the course of the study: – indian diabetic retinopathy image dataset (idrid) – digital retinal images for vessel extraction (drive) the strategy employed for data collection was sourcing images from the indian diabetic retinopathy image dataset (idrid), the first dataset of this kind that is typical of the indian populace. moreover, it is the only dataset featuring pixel-level observations of typical diabetic retinopathy lesions and normal retinal frameworks. a retinal doctor at an eye center in nanded, maharashtra, india took the fundus photos. the images were taken with a 50-degree field of view kowa vx-10 alpha digital fundus camera (fov) and were verified by experts to be of adequate quality, clinical relevance, and containing a decent combination of disease grades indicative of diabetic retinopathy (dr). of the thousands of examinations available, 516 images with a resolution of 4288x2848 pixels were chosen and saved in jpg format with an average size of roughly 800 kb. expert annotations of typical diabetic retinopathy impairment and normal retinal components are included in the dataset. it also provides information regarding the severity of diabetic retinopathy and diabetic macular edema for each picture in the database, based on global clinical relevance criteria. the image data was downloaded as a compressed zip folder from the source database to a computer system. the 516 samples were divided into 413 and 103 training and testing samples respectively. each image in the dataset is annotated with a label in the range of 0 to 5, which represents the diabetic retinopathy severity class corresponding to the image. the digital retinal images for vessel extraction (drive) dataset is a dataset for retinal vessel segmentation. this consists of a total of 40 color fundus images; including 7 showing mild proportions of diabetic retinopathy and the other 33 images showing no presence of the disease. the images are divided into 20 training and 20 test images with corresponding image masks. the images were obtained from a diabetic retinopathy screening exercise in the netherlands. a canon cr5 non-mydriatic 3ccd camera with a 45-degree field of view was used to capture the photographs (fov). at 768 by 584 pixels, each image was taken utilizing an 8-bit-per-color plane. each image has a circular field of view (fov) with 540 pixels diameter. the cropped images near the fov were used for this database. a delimited fov image mask is provided for each image. the drive database was created to allow for comparative studies on blood vessel segmentation in retinal images and has proven useful in several studies in this regard. 3.2 data preprocessing the proposed method outlines the steps involved in transforming the data before feeding it into the image segmentation and siamese neural networks. image augmentation: achieving high performance on convolutional neural networks often requires having sufficient data samples for training. hence, several image augmentation techniques were applied to the drive dataset which contains only 20 training samples to increase the training sample size. the following augmentation techniques in the argumentation library in python were employed for the dataset thereby increasing the sample size to 120. – horizontal flip – vertical flip – elastic transform – grid distortion paradigmplus (2023) 4:1 continuous eye disease severity evaluation 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . – optical distortion during the augmentation, extra attention is paid to make sure that the masks are getting the same changes that are being applied to the images. histogram equalization: according to reza [21], histogram equalization is a relatively basic approach for image improvement as captured real-time series of images, in their true form may lack excellent viewing quality owing to the absence of adequate lighting or intrinsic noise. a notable method of interest that is widely used for the enhancement of still images is contrast limited adaptive histogram equalization (clahe). clahe is an enhanced version of adaptive histogram equalization (ahe). both surmount the limitations of standard histogram equalization. the algorithm is used to partition the images to be applied to the siamese network into contextual sections and histogram equalization is applied to each one. this levels off the distribution of used grey pixel values and thus makes concealed features of the image more glaring. the complete grey spectrum is used to show the image. rotation/resizing/mirroring/brightness (for too-dark images): in this pre-processing stage, random rotation (range of 10°), horizontal flipping (50% probability), random pixel cuts (to 512 × 512 pixels), and random brightness and contrast variation (range of 3%) were applied to the photos. for validation and testing purposes, images were center cropped (to 512×512 pixels). every color image was converted to a grey scale. 3.3 image segmentation image segmentation is a technique in image processing that partitions an image into regions based on particular features and properties. it is aimed at making the image easier for analysis and achieving better results in computer vision tasks. to achieve the required granularity and specificity required for determining severity using the triplet network, a convolutional neural network built with the u-net architecture was trained to perform retinal vessel segmentation. joshua et al., [22] pointed out that for precise and speedy detection of eye illnesses such as glaucoma, diabetic retinopathy, and macular degeneration, proper examination of retinal fundus images is critical. u-net was chosen to perform image segmentation for the following reasons: – it has been proven to perform well when applied to image segmentation tasks since it performs image localization by predicting the retinal images pixel by pixel. – u-net does not require a large amount of training data, making it perfect for biomedical image analysis because of the relative paucity of images in the field of biomedicine. the basic u-net architecture employed is shown in figure 3. after image augmentation was used to significantly increase the data set, the images were sorted and passed to the u-net model for training. the network was trained on a batch size of 2 and for 50 epochs with 55 steps per epoch. afterward, the model is evaluated for important performance metrics. 3.4 siamese network architecture three identical subnetworks with shared weights were used to build a siamese neural network using a triplet loss function. each subnetwork is a pre-trained resnet-101 convolutional neural network trained on the imagenet dataset and all networks are joined at the output. figure 4 shows the siamese network. paradigmplus (2023) 4:1 8 muyideen abdulraheem et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 3: u-net architecture figure 4: siamese network 3.5 triplet mining algorithm this algorithm was used to determine the selection of triplet samples based on the severity class of the annotated anchor image before being fed to the siamese network. given the triplet architecture of the siamese network, the data is divided into triplet image pairs (anchor, positive and negative images) based on the proposed algorithm. paradigmplus (2023) 4:1 continuous eye disease severity evaluation 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 severity evaluation method the input to the siamese network architecture is a pair of three images, an anchor image, a positive image (similar to the anchor), and a negative image (dissimilar from the anchor). each image pair is passed through the pre-trained network which seeks to learn the distributed embedding of the images by the notion of similarity and dissimilarity to the anchor image. the anchor (a), positive (p), and negative (n) images are passed through their respective convolutional neural networks, and during backpropagation weight vectors are updated using the shared architecture. the euclidean distance between the images is then calculated from the final connected layer. this distance is representative of the difference between the images and is abstracted to represent the severity score for the image. 3.7 model deployment the goal of building any system is to make it available to the end user, deep learning models are no exception. model deployment is one of the final and most challenging steps in any machine learning workflow. models are usually deployed via api, web, or app interfaces which users can interact with for their intended use cases. the study employs flask which is a web application framework to deploy the u-net and siamese networks for use in production. flask’s variety of tools and libraries make it easy to integrate machine learning models into its api. 3.8 performance metrics these metrics are used to evaluate the performance of trained models. in this study, we evaluate the performance of two models; the u-net model used for retinal vessel segmentation and the triplet siamese model. the u-net model was evaluated on the: – accuracy: model accuracy is a metric used to determine which model performs best at recognizing correlations and patterns between variables in a dataset based on the input, or training, data (see equation 1, where tp is truly positive, f p is false negative, tn is true negative). a = tp + tn tp + tn + f p + f n (1) – precision: model precision measures what proportion of positive predictions was correct. it is defined as the number of true positives divided by the sum of the number of true positives and false positives (see equation 2, where tp is truly positive, f p is false negative). p = tp tp + f p (2) – recall/sensitivity: the recall of a model is a measure of how well the model detects true positives. it is the total number of cases correctly predicted as positive of the total positive cases. for this use case, recall measures the ability of our u-net to correctly identify the location of white lines drawn for the mask generated from each eye fundus image (see equation 3, where tp is truly positive, f p is false negative). se = tp tp + f n (3) furthermore, the siamese model used in this research is evaluated based on the correlation between its output and the disease severity class of the fundus images. all processes involved in the paradigmplus (2023) 4:1 10 muyideen abdulraheem et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . research, including image preprocessing and model training, were carried out on a macbook pro (2015) with a 2.7 ghz dual-core intel core i5 processor having a random-access memory (ram) of 8 gigabytes. all development efforts were made in python. python is an interpreted multipurpose programming language. the substantial design philosophy used stresses code readability through the use of indentation, and its language components and object-oriented paradigms are aimed at assisting programmers in writing clear and logical code for both small and big projects. python is the preferred programming language for data science tasks because it is simple to use, has good library availability, active community engagement, and a dynamic selection of libraries and resources. inactive communities are less likely to maintain or develop their platforms, which is not the case with python. the principal python libraries used in the course of the implementation were pandas, pytorch, tensorflow/keras, scikit-learn, and flask. 3.9 why the siamese neural network the siamese neural network was chosen for this investigation because of its capacity to predict better with fewer input data. siamese networks have grown in popularity in recent years due to their capacity to learn from relatively little input. with the help of one-shot learning, a few images per class are enough for siamese networks to recognize images. siamese learning requires a small number of parameters (weights), hence training the network will likely converge faster and the network will be less volatile to noise. using siamese to build the model avoids the model computing the output for the input just once, and then using the cached results for the existing images in the database to quickly compute the similarity measures. snn focuses on learning embeddings (in the deeper layer) that group comparable classes/concepts together and hence decide on semantic similarity. also, siamese networks learn from semantic similarity rather than training mistakes or mispredictions. this helps the model to acquire even better embeddings that represent images from the support set and bring related ideas closer together in the feature space. instead of only extracting static features using convolutions, the model learns ideas and seeks to comprehend why certain images are more similar than others by learning such a feature space, similar to how textual models learn word embeddings. the capacity to provide benchmark performance on extraordinarily little data is the most immediately relevant advantage. the problem of class imbalance disappears when the data need is minimized. 4 results analysis the experimental results are presented for the model employed for image segmentation as well as the siamese network outputs. 4.1 results of training and testing u-net model the results of the prediction of a u-net model which was trained on 120 retinal fundus images using dice loss: accuracy 95%, precision 38%, recall 95%, f1 score 55%. the high accuracy (95%) of the model shows its overall ability to correctly segment the retinal vessels from well-captured retinal fundus photographs. however, the low precision (38%) value results from the fact that image segmentation tasks are not as straightforward as linear prediction tasks largely because image segmentation tasks require that the test image is properly captured, including the presence of proper lighting conditions and powerful lens cameras. therefore, the precision value here would be a result of not being able to capture the image structure for certain poorly taken images. the high recall value (95%) implies that of the properly taken retinal images, the model could accurately produce a closeto-identical image mask for the larger number. this indicates that retinal fundus images taken with high-resolution cameras which produce fundus images with visibly seen vessels can perform well paradigmplus (2023) 4:1 continuous eye disease severity evaluation 11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 5: u-net fundus image and mask comparison on this model. figure 5 shows the comparison of the image masks generated by the trained u-net models and the original annotated masks from the test set. 4.2 severity evaluation this section of the work presents key findings about eye disease severity determination using trained siamese neural networks on single images and images taken for the same patient at different time points. 4.2.1 evaluation of disease severity for single images disease severity was evaluated for images taken at single time points by comparing the single image(anchor) to a pool (5) of randomly selected dissimilar (negative) images following the triplet mining algorithm described earlier. first, the single image is passed through the u-net model described above then both anchor and negative images are passed through a siamese network comprised of resnet-101 models which generate an image embedding for each image. the siamese neural network is used to compute the euclidean distance between the image pairs. the median of these computed euclidean distances is interpreted as the measure of disease severity. these euclidean distances are plotted against their numerical severity grade in the figure above. the findings in figure 6 shows a correlation between the euclidean distance of the image embeddings and the grade of disease severity assigned for the image. this implies the following: – that the severity score generated is a valid representation of eye disease severity. – image segmentation enables the siamese network to achieve better specificity in severity prediction. 4.2.2 evaluation of disease severity change longitudinally evaluating a pair containing two images taken from a particular patient at different instances of time to determine the severity of disease and the change on a continuous scale occurs in two steps: paradigmplus (2023) 4:1 12 muyideen abdulraheem et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0,005 0,01 0,015 0,02 0,025 0,03 0,035 0,04 0,045 0,05 0,055 0 0,5 1 1,5 2 2,5 3 3,5 4 4,5 f(x) = 94,9579831932773 x − 0,605042016806722 r² = 0,901701857213473 correlation between siamese network output and severity grade figure 6: scatter plot between the siamese network output and severity – two images in the pair are inputted directly to the network which computes their euclidean distance representing disease severity. – the euclidean distance for every single image in the pair is calculated using the earlier proposed algorithm. the computed difference between these euclidean distances is then used to quantify disease severity and change. from the scatter plot in figure 6, – it is observed moderate correlation between the longitudinal disease severity change between two images taken at different time points and the difference between the severity scores generated for both images using the earlier proposed method. – this demonstrates that disease severity change in the same patient can reasonably be measured over time by finding the distance between any two image samples computed by the siamese network. 5 model deployment the proposed system is deployed via a flask web application providing a comfortable interface for medical practitioners to work with. the web application is deployed to a production environment via heroku for ease of access. the flask web application provides the following features: – an interface where users can navigate between single disease severity evaluation and, multipoint disease severity evaluation. paradigmplus (2023) 4:1 continuous eye disease severity evaluation 13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 7: web evaluator home page figure 8: web app fundus image upload – functionality that enables users to upload retinal fundus images to get a severity score back as a response. figures 7 and 8 show key features of the application. 6 future work we anticipate a future in which continuous disease severity ratings may be utilized to more consistently quantify disease severity and follow progress over time. this might be useful in ordinary health paradigmplus (2023) 4:1 14 muyideen abdulraheem et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . treatment as well as clinical research. prior work on deep learning classification has investigated the benefit of adopting a continuous measure for performance evaluation to increase model accuracy, however, that classification approach ultimately predicts ordinal illness grades. using measurements obtained from euclidean distance output, our siamese neural network technique presents a potentially generalizable solution for producing continuous disease severity grading scales for medical pictures. this work has a number of limitations. initially, the training labels were constructed from the ordinal illness severity class labels. instead of labels assigned by expert comparisons of pictures, future research may integrate labels assigned by expert comparisons of images for training. in addition to potentially enhancing training performance, the comparison labels might be used with ordinal illness severity class labels for training, thus reducing the number of pictures that need to be annotated. second, we established the applicability of our method to two-dimensional imaging issues, despite the fact that typical medical imaging requires three-dimensional data. further research will need to look at the applicability of this method to volumes. finally, the models in this work were trained using a contrastive loss function on the euclidean distance computed from the last completely connected layers. other techniques, such as evaluating the loss function on earlier network layers or using totally different loss functions, such as the triplet loss function, margin rank loss, and other changes to the training methodology, may increase model performance. alternative loss functions may include more information in the labels, such as magnitude and directionality, which might possibly boost learning even more. 7 conclusions the study demonstrated the use of siamese neural networks to evaluate eye disease severity on a continuous scale. the wide range of literature reviewed in the course of this research revealed the majority of efforts geared towards evaluating disease severity evaluation have been done by employing machine learning models generating discrete severity categories, it also identified the failings of these systems. the study demonstrated the utility of employing image segmentation before disease severity evaluation and leveraging the ability of pre-trained models. the research met its key objectives meets its key objectives by demonstrating in two scenarios that evaluating eye disease severity quantitatively better represents its true underlying continuous nature. subsequent research can train models with the use of data with expertly assigned severity grades, thus limiting the subjective input of medical practitioners. severity scores generated can be used automatically to reconstruct images that would predict the progression of the disease given the severity change across time. investigating the applicability of this approach to 3-dimensional images which are increasingly prevalent in medical imaging and radiography. authors’ information – muyideen abdulraheem is is a lecturer i in the department of computer science, university of ilorin, ilorin, nigeria. he obtained a bachelor of technology computer science at the prestigious abubakar tafawa balewa university bauchi, in 1997, a master of science in mathematics (computer option), and a ph.d. computer science at the university of ilorin, nigeria. his research area is computer security, grid computing security, and data and information security. – idowu d. oladipo received his b.sc.(edu.) degree in computer science from ekiti state university, ado-ekiti, ekiti state, nigeria in 2005. he earned his m.sc. and ph.d. degrees in computer science from the university of ilorin, nigeria, in 2010 and 2018 respectively. since 2019, he has been a lecturer ii with the department of computer science, university of ilorin, ilorin, nigeria. his research interests include software engineering, bioinformatics, information security, paradigmplus (2023) 4:1 continuous eye disease severity evaluation 15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . artificial intelligence, cyber security, and computer education. he is a member of the international computer professional registration council of nigeria (mcpn) and nigeria computer society (mncs). – sunday adeola ajagbe is a faculty member at the computer and industrial production engineering department, first technical university, ibadan, and a ph.d. candidate at the computer engineering department, ladoke akintola university of technology (lautech), both in nigeria. he obtained a master of philosophy in the computer engineering department at lautech, msc and bsc in information technology and communication technology respectively at the national open university of nigeria (noun), and his postgraduate diploma in electronics and electrical engineering at lautech. he specializes in artificial intelligence (ai), natural language processing (nlp), information security, data science, the internet of things (iot), sensors, communications, and signal processing. he is also licensed by the council regulating engineering in nigeria (coren) as a professional electrical engineer, and a student member of the institute of electrical and electronics engineers (ieee), and the international association of engineers (iaeng). – ghaniyyat b. balogun received her b.tech. degree in mathematics/computer science from the federal university of technology, minna, nigeria in 1998. she obtained a master’s degree in business administration (mba), together with an msc. in mathematics/computer science and phd. in computer science degrees from the university of ilorin, nigeria in 2003, 2008, and 2015 respectively. she is presently a lecturer in the computer science department of the university of ilorin, nigeria. her research interests fall in the areas of data structures, big data, data mining, data analytics, and database management systems. she is a member of the international computer professionals registration council of nigeria (mcpn), nigeria computer society (ncs), the international association of engineers (ieang), the institute of strategic management of nigeria (ismn), nigerian women in information technology (niwit) among others. – mulikat bola akanbi is a lecturer at the department of computer science, institute of information and communication technology, kwara state polytechnic, ilorin, nigeria. she obtained her b.sc and m.sc in computer science from the university of ilorin in 2003 and 2014 respectively. she is currently a ph.d. student in the same university. her research interests include information security, computer networks, data mining, biometric authentication, and template protection. she is a professional member of the nigeria computer society (ncs), academia in information technology (aitp), nigeria women in information technology (niwiit), and women in technical education and development (wited). – nissi o. emma-adamah is a graduate of computer science from the university of ilorin, nigeria. he graduated in the year 2022. authors’ contributions – muyideen abdulraheem participated in conceptualization, review, validation, and supervision. – idowu d. oladipo participated in the writing of the original draft, methodology, and coding. – sunday adeola ajagbe participated in the methodology, resource management, writing of the original draft, editing, and supervision – ghaniyyat b. balogun participated in resource management, project administration, and visualization. paradigmplus (2023) 4:1 16 muyideen abdulraheem et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . – mulikat bola akanbi participated in resource management and editing. – nissi o. emma-adamah participated in editing, review and supervision. competing interests the authors declare that they have no competing interests. funding no funding was received for this project. references [1] c. p. wilkinson, f. l. ferris iii, r. e. klein, p. p. lee, c. d. agardh, m. davis, d. dills, a. kampik, r. pararajasegaram, j. t. verdaguer, et al., “proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales,” ophthalmology, vol. 110, no. 9, pp. 1677– 1682, 2003. [2] a. colomer, j. igual, and v. naranjo, “detection of early signs of diabetic retinopathy based on textural and morphological information in fundus images,” sensors, vol. 20, no. 4, p. 1005, 2020. [3] s. a. ajagbe, k. a. amuda, m. a. oladipupo, f. a. oluwaseyi, and k. i. okesola, “multiclassification of alzheimer disease on magnetic resonance images (mri) using deep convolutional neural network (dcnn) approaches,” international journal of advanced computer research, vol. 11, no. 53, p. 51, 2021. [4] m. d. li, n. t. arun, m. gidwani, k. chang, f. deng, b. p. little, d. p. mendoza, m. lang, s. i. lee, a. o’shea, et al., “automated assessment and tracking of covid-19 pulmonary disease severity on chest radiographs using convolutional siamese neural networks,” radiology: artificial intelligence, vol. 2, no. 4, p. e200079, 2020. [5] a. b. rosenkrantz, r. duszak jr, j. s. babb, m. glover, and s. k. kang, “discrepancy rates and clinical impact of imaging secondary interpretations: a systematic review and meta-analysis,” journal of the american college of radiology, vol. 15, no. 9, pp. 1222–1231, 2018. [6] j. p. campbell, j. kalpathy-cramer, d. erdogmus, p. tian, d. kedarisetti, c. moleta, j. d. reynolds, k. hutcheson, m. j. shapiro, m. x. repka, et al., “plus disease in retinopathy of prematurity: a continuous spectrum of vascular abnormality as a basis of diagnostic variability,” ophthalmology, vol. 123, no. 11, pp. 2338–2344, 2016. [7] m. kim, j. yun, y. cho, k. shin, r. jang, h.-j. bae, and n. kim, “deep learning in medical imaging,” neurospine, vol. 16, no. 4, p. 657, 2019. [8] s. a. ajagbe, o. a. oki, m. a. oladipupo, and a. nwanakwaugwum, “investigating the efficiency of deep learning models in bioinspired object detection,” in 2022 international conference on electrical, computer and energy technologies (icecet), pp. 1–6, ieee, 2022. [9] j. bromley, i. guyon, y. lecun, e. säckinger, and r. shah, “signature verification using a" siamese" time delay neural network,” advances in neural information processing systems, vol. 6, 1993. [10] d. chicco, “siamese neural networks: an overview,” artificial neural networks, pp. 73–94, 2021. paradigmplus (2023) 4:1 continuous eye disease severity evaluation 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [11] d. s. w. ting, l. r. pasquale, l. peng, j. p. campbell, a. y. lee, r. raman, g. s. w. tan, l. schmetterer, p. a. keane, and t. y. wong, “artificial intelligence and deep learning in ophthalmology,” british journal of ophthalmology, vol. 103, no. 2, pp. 167–175, 2019. [12] m. r. k. mookiah, u. r. acharya, c. k. chua, c. m. lim, e. ng, and a. laude, “computer-aided diagnosis of diabetic retinopathy: a review,” computers in biology and medicine, vol. 43, no. 12, pp. 2136–2155, 2013. [13] u. r. acharya, c. m. lim, e. y. k. ng, c. chee, and t. tamura, “computer-based detection of diabetes retinopathy stages using digital fundus images,” proceedings of the institution of mechanical engineers, part h: journal of engineering in medicine, vol. 223, no. 5, pp. 545–553, 2009. [14] k. a. anant, t. ghorpade, and v. jethani, “diabetic retinopathy detection through image mining for type 2 diabetes,” in 2017 international conference on computer communication and informatics (iccci), pp. 1–5, ieee, 2017. [15] g. mushtaq and f. siddiqui, “detection of diabetic retinopathy using deep learning methodology,” in iop conference series: materials science and engineering, vol. 1070, p. 012049, iop publishing, 2021. [16] j. k. adeniyi, a. e. adeniyi, y. j. oguns, g. o. egbedokun, k. d. ajagbe, p. c. obuzor, and s. a. ajagbe, “comparison of the performance of machine learning techniques in the prediction of employee,” paradigmplus, vol. 3, no. 3, pp. 1–15, 2022. [17] m. leeza and h. farooq, “detection of severity level of diabetic retinopathy using bag of features model,” iet computer vision, vol. 13, no. 5, pp. 523–530, 2019. [18] m. u. akram, s. khalid, a. tariq, s. a. khan, and f. azam, “detection and classification of retinal lesions for grading of diabetic retinopathy,” computers in biology and medicine, vol. 45, pp. 161–171, 2014. [19] n. b. prakash, d. selvathi, and g. r. hemalakshmi, “development of algorithm for dual stage classification to estimate severity level of diabetic retinopathy in retinal images using soft computing techniques.,” international journal on electrical engineering & informatics, vol. 6, no. 4, 2014. [20] s. taylor, j. brown, k. gupta, j. campbell, s. ostmo, and r. chan, “imaging and informatics in retinopathy of prematurity consortium. monitoring disease progression with a quantitative severity scale for retinopathy of prematurity using deep learning,” jama ophthalmol, vol. 137, no. 9, pp. 1022–1028, 2019. [21] a. m. reza, “realization of the contrast limited adaptive histogram equalization (clahe) for realtime image enhancement,” journal of vlsi signal processing systems for signal, image and video technology, vol. 38, pp. 35–44, 2004. [22] a. o. joshua, f. v. nelwamondo, and g. mabuza-hocquet, “blood vessel segmentation from fundus images using modified u-net convolutional neural network,” journal of image and graphics, vol. 8, no. 1, pp. 21–25, 2020. paradigmplus (2023) 4:1 introduction related work methodology data collection data preprocessing image segmentation siamese network architecture triplet mining algorithm severity evaluation method model deployment performance metrics why the siamese neural network results analysis results of training and testing u-net model severity evaluation evaluation of disease severity for single images evaluation of disease severity change longitudinally model deployment future work conclusions vol. 2, no. 1, pp 20 39, 2021 doi: 10.55969/paradigmplus.v2n1a2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . implementation of an irrigation system to estimate water in hydroponic crops through evapotranspiration jorge hernandez id 1,� and karen daza id 2 1universidad distrital francisco jose de caldas, bogota, colombia hrjorgee@correo.udistrital.edu.co kgdazaa@correo.udistrital.edu.co abstract hydroponic crops are a special type of crop because it tries to reduce space and water use for certain plants. normally, the estimation of the water of the crops in underdeveloped countries is made in a manual way generating waste of water. in the water estimation, ideal factors can be considered in which the crops carry out their biological process with a high yield. the hydric estimation can be measured as a function of the soil, the ambient temperature, the diameters of the roots, and the interspace between the plants. we consider carrying out the water estimation by checking each plant to reach the correct progress of plants in the crop. we show an approach to integrating hardware and software components. the prototype is capable of calculating the evapotranspiration of crops and making a water estimation according to the several factors in each plant and the measured data from electronic sensors in order to reduce water in each irrigation. additionally, the system allows the farmers to schedule the irrigation. keywords: water estimation ⋅ crop irrigation ⋅ irrigation scheduling ⋅ evapotranspiration received: 25 february 2021 ⋅ accepted: 16 april 2021 ⋅ published: 29 april 2021. 1 introduction agriculture plays a fundamental role in the economic development in agricultural countries, where its main function is to provide food and the necessary resources for other activities. the modernization, the study, and the contributions made to this field are of great importance so that the undeveloped countries carry out the production and exportation of products. this work carried out seeks to contribute to this branch, saving water and analyzing each plant in the crop. controlling the use of water in crops allows for better crop performance [1]. in undeveloped countries, most crops do not have a smart irrigation system to control the water estimation, which increases the probability of a manually hydric estimation. then, crop irrigation is performed without taking into account the needs and ideal factors for each plant, such as evapotranspiration, temperature, humidity, etc. the evapotranspiration is calculated with the water evaporation and transpiration. manual irrigation can generate a lot of waste since irrigation is done with a hose sporadically. depending on the plant species and crop diseases, the amount of water in each plant tends to vary. sometimes it is the soil that absorbs most of the liquid instead of the plants since the plants need a certain time to consume the water. this article is distributed under the terms of the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. https://doi.org/10.55969/paradigmplus.v2n1a2 https://orcid.org/0000-0002-7129-7919 https://orcid.org/0000-0003-4555-0199 implementation of an irrigation system to estimate water in hydroponic crops through evapotranspiration 21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . generally, hand watering is done until the soil has a large amount of water. however, depending on the type of plant the water irrigation must be equilibrated, several studies show the analysis of crops variating the water [2] [3], in order to establish a goodness result in the crop, es very important to calculate the water. depending on the species, some plants consume water faster than others. in most cases, hydroponic crops are grown in urban spaces, in some cases, people do not have the time or the knowledge to make a water estimation; in some cases, water is simply applied without taking into account the ideal factors of the plants and the soil. the water resource in a country becomes essential when there is not a good distribution of soil or space [4]. thus, the hydroponic crops come about in order to perform the use of soil and space in the urban. in this work, we carry out the implementation of a prototype using components of software and electronic devices. the prototype can perform the function of estimating and watering the plants, measuring the different factors, this is done by moving the device to each of the plants that are in the crop and obtaining the required values, which will allow us to calculate the necessary water. to solve cost issues, the prototype is transported to each plant and the pertinent measurements are made, thereby saving the purchase of sensors per plant. an interface was created so that the farmer in charge can remotely control the device and manage the irrigations. we use the tomato crop as a case study to use the prototype implementation. in addition, this article is an extension of this paper [5] with more specifications in the water estimation. the paper is structured as follows. section 1 presents the introduction. section 2 shows the concepts in the water estimation. in section 3 we present the related work. in section 4, we explain the approach of this work. in section 5, the results are explained. section 6 shows a discussion. finally, in section 7 the paper concludes. 2 hydroponic crops in hydroponic crops, there are different methods to replace the soil that are normally used in normal crops. on the other hand, it is very important to keep in mind the water estimation in order to save water. using this type of crop reduces water use by a considerable percentage, approaching 100%. also, plant nutrients are not lost but, on the contrary, are continuously stored and recycled. in addition, herbicide use is non-existent and pesticides are natural. [6]. normally, hydroponic crop irrigation is made by artificial methods or using a design of water pipeline to achieve multiple irrigations in the plants. without any use of irrigation, the crops are subjected to the availability and reliability of naturally occurring water from floods or rain. depends on the type of plants, but, in the hydroponic crops, the plants are not exposed to rain instead using pipeline irrigation. there are different types of irrigation methods and techniques used for watering crops. for example, by drip, sprinkler, and microinjection. each one has a characteristic that makes it the most effective method according to the crop features. nevertheless, in most cases, the drip irrigation method requires less water than others [7]. 2.1 evapotranspiration (et) evapotranspiration is a process that is composed of two sub-processes, which are evaporation and transpiration, each of these two sub-processes being carried out separately. et is calculated to know exactly how much water is lost to the atmosphere. on the one hand, evaporation is the process where liquid water is converted to steam and removed from the evaporating surface. the water evaporates from surfaces such as rivers, lakes, vegetation, or even the pavement. on the other hand, transpiration consists of the vaporization of liquid water contained in plant tissues [8]. et is expressed in millimeters (mm) per unit of time and indicates the water lost in a cultivated area. table 1 shows the factors that influence the et, which are the climatic variables, the types of crops, and the environmental conditions of the land. paradigmplus (2021) 2:1 22 jorge hernandez and karen daza . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 1: factors influencing et factor description climatic variables radiation, air temperature, atmospheric humidity, and wind speed. crop to estimate evapotranspiration, the type of crop used and the stage of development in which it is found must be taken into account. ambient conditions the inadequate use of essential fertilizers, the parasites found in the soil, and the decrease in its fertility limit the correct development of the crop. reference crop et (eto) is the rate of evapotranspiration from a hypothetical reference crop with an assumed crop height of 0.12 m (4.72 in), a fixed surface resistance of 70 sec m−1 (70 sec 3.2f t−1) and an albedo of 0.23, closely resembling the evapotranspiration from an extensive surface of green grass of uniform height, actively growing, well-watered, and completely shading the ground [9]. in addition, the relation between eto and et with a certain surface allows making the relation with other surfaces by removing the process of defining the et level for each crop and period. then, it is important to study the rate of et in the atmosphere regardless of the type of crop and stage considering that the crop does not have water stress or diseases. the following concepts are required for the calculation of eto. 1○ average temperature of the environment (thr). this value is obtained by calculating the average between the maximum and minimum air temperature during 24 hours of constant observation. 2○ saturation pressure of steam at air temperature (e0(thr)). the pressure generated in the gas phase is known as saturation, which occurs when the temperature of the liquid water and the vapor remain at a constant equilibrium. 3○ saturation pressure of steam (es). it occurs when we obtain the average result of the saturated vapor pressure with a maximum temperature and the same pressure but at a much lower temperature, this with respect to air. 4○ relative humidity (rh). this process is obtained by dividing the amount of water that the air would contain at a given temperature by the amount of water it could contain if it were saturated at the same temperature. 5○ actual steam pressure derived from the temperature of the dew point (td). this process occurs when the dew temperature is at its point since this process is the appropriate temperature at which the air must be cooled until it is saturated. 6○ slope of steam pressure curve (∆). it is the ratio of saturated vapor pressure to temperature. 7○ atmospheric pressure (p): during this process, the earth’s atmosphere exerts pressure on the given weight, which is expressed in kilopascals (kpa). 8○ latent heat of vaporization (lv): it is known as the energy used to convert the mass of liquid water into vapor at constant pressure and constant temperatures. 9○ psychometric constant (γ): it is air as a mixture of two gases that do not interact with each other and behave as two gases that are ideal, which are defined as dry air and water vapor. paradigmplus (2021) 2:1 implementation of an irrigation system to estimate water in hydroponic crops through evapotranspiration 23 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10○ wind speed in relation to height (u2): this process is used to calculate evapotranspiration and thus to define how it varies over time, for this, it is necessary to express it as the average of a defined interval in time as a function of time velocity. 11○ extraterrestrial radiation (ra): this variable is obtained on a surface that is at the top of the atmosphere when it receives daily solar radiation. 12○ solar radiation (rs): it is the process by which hydrogen reactions are obtained, emanated by the nucleus of the sun during nuclear fusion, and thus emitted to the solar surface, all this is obtained from the energy coming from the sun. 13○ net solar radiation or long-wave (rln ): the amount of long-wave energy emission proportional to the absolute surface temperature is raised to the fourth power. the net flux of energy leaving the earth’s surface is less than that calculated and given by stefan boltzmann’s law due to absorption and reflected radiation from the sky. 14○ net radiation (rn): is the process by which all the fluxes obtained from the radiation are added up, there are two values, short wave, and longwave, also taken from those coming from the surface. 15○ density of soil heat flow (g): in this process, the amount of energy transferred through the soil is measured, it is estimated when the soil heats up, as this depends on the net radiation. et of the crop under standard conditions (etc) is the calculation of the et of any crop when it has optimal soil conditions, abundant fertilizer, and without diseases. it is calculated from climatological data. to make irrigation systems in hydroponic crops, it is necessary to analyze the space and the type of plant. the water requirement required for a crop will depend on both the baseline et rate pattern and the distribution of the crop coefficient curves for the crops of interest during the growing season [10]. 2.2 hydric estimation in water estimation, is necessary to know the inputs and outputs in order to subtract the outputs and added the inputs. we used four inputs in the water estimation: previous irrigation, precipitation, capillary rise, and surface drainage. to calculate the water in the irrigation we based on the system dssat1 and models of water use for crops [11, 10] respectively. 3 related work according to our statement of the problem, we search articles where: 1. articles that discuss electronic prototypes that determine the specific amount of irrigation per plant. 2. articles that present prototypes to schedule the irrigation with a graphical interface to the farmers. the related work is presented as follows. gocic et al., [12] created software in order to estimate the et of reference using weather information. the authors present a user-friendly software that helps to calculate the et0. the authors did not present a hardware device to generate the irrigation in the crops. zhang et al., [13] developed a scheme that brings together hardware and software, in order to calculate et in real-time and through different processes to define automatic irrigation, based on the crop period and climatic aspects. to obtain the information, a hardware component was implemented that sends the recorded data to a server where irrigation is managed. the device used to irrigate the 1https://dssat.net/ paradigmplus (2021) 2:1 24 jorge hernandez and karen daza . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . crop is in charge of determining the right moment to irrigate, on the other hand, it performs the calculations corresponding to the water balance, with the et obtained. the approach of our project validates the factors individually in each plant, thus obtaining more accurate values. shah et al., [14] built a wireless sensor network (wireless sensor networks) to be implemented in a grape crop. the objective of this research is to strengthen the requirements for optimal irrigation, in order to contribute to water saving. the sensors were placed in an area of 30m x 30m. each of these sensors could receive and send information to each other within a defined distance of 30m. the network of sensors is responsible for obtaining data on soil moisture and ph, as well as collecting the ambient temperature. with this information collected, a node defined as the main one is in charge of making decisions in the defined area. in our research, the fact of implementing a sensor network would not be easy because of the type of crop, so our approach implements sensors that move to each of the crop plants. umair et al., [15] describe the implementation of a neural network that can control an irrigation method. sensors that are installed in the crop obtain the inputs. the objective of such a method is to be able to maintain adequate moisture, taking into account the real-time moisture collected from the soil. finally, they mention that the capabilities of the network to control are not as expected, which becomes a limiting factor to successfully completing the implementation. shamshirband et al., [16] use the cuckoo search algorithm (csa) to optimize the estimation from the artificial neuronal network (ann) and the adaptive neuro-fuzzy inference system (anfis) models to calculate the evapotranspiration. the authors compared the algorithms to calibrate the evapotranspiration equation. the authors did not discuss the implementation of irrigation in any crop. suciu et al., [17] developed an irrigation system that collects information on air, temperature, plant evapotranspiration, soil moisture, frequency of precipitation, wind direction, and relative pressure. such a system is able to control the irrigation systems according to the time determined by the information previously collected from the various sensors. but it is still up to the farmer to decide how much water to irrigate. prisilla et al., [18] discuss other irrigation systems that use the fixed rate for irrigation and do not use a necessary quantity of water for the crop. the authors created a system using an artificial neural network (ann) with the next parameters: air, temperature, soil moisture, radiations, and humidity to calculate the water in a rice crop. the ann model also requires evapotranspiration to generate the water estimation. the authors only crated hardware to control the irrigation, the farmer is not interactive with the system. 4 proposed approach the proposed system is integrated with several components that communicate to estimate water and schedule irrigation. the software components have the function of controlling the sensors, managing the data on the server, and presenting the views to the user so that the farmer can schedule the irrigations. the hardware elements are mainly sensors, devices to manage the water in the plants, and devices to move to each plant. 4.1 components of system in the proposed system each component has a unique function. in some cases, the component must communicate with another component. the interaction of the system components can be seen in figure 1. the central system and the client application are software components. the first one allows managing the database and calculating the evapotranspiration. the second one is a visual interface to farmer interaction. on another hand, the master component and slave component are electronic components. the first one is the handler of the next operations: irrigation, sensing the soil/temperature, and moving to a certain plant. the second one is the device that makes the action paradigmplus (2021) 2:1 implementation of an irrigation system to estimate water in hydroponic crops through evapotranspiration 25 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . client aplication web request web browser operating system client system web app resources central system data management cloud bd api rest handle requests schedule irrigation schedule temperature by hour http master component in router bluetooth device master processor system controller cpu slave componentin crop slave processor manage operation system controller cpu execute operation sensors humidity temperature bluetooth device actuators axis x axis ybluetooth axis z electrovalve tcp/ip ethernet component http request figure 1: system architecture that the master component sends it. next, the four main components in the proposed system are explained. 1○ central system. the objective of this component is to store the system data and generate the irrigations that have been programmed by the farmer. this component has different web services. the web services are mainly for the client application component and the master component. the central system component was made with the nodejs 2 language and mongodb3 was used to store the data. mongodb is a non-relational database. mainly, the central system component has two subcomponents: a) rest api. it is responsible for managing http requests, programming temperature measurements (to calculate eto), and generating scheduled irrigations. b) cloud db. it is a database in mongodb. the rest api and cloud db subcomponents connect over tcp/ip. 2○ client application. this component was made with vuejs4. the main objective of this component is to generate the views so that the farmer can interact with the system. the main actions that the farmer can carry out are: to view the water used in the crop, program irrigation, create plants and apply irrigation. this component communicates with the central system component using web services and the http protocol. 2https://nodejs.org/en/ 3https://www.mongodb.com/ 4https://vuejs.org/ paradigmplus (2021) 2:1 https://nodejs.org/en/ https://www.mongodb.com/ https://vuejs.org/ 26 jorge hernandez and karen daza . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3○ master component. this component is an arduino that is on the crop but must be connected to the router to access an internet connection. the main objective of this component is to review if any operation should be carried out on the crop. to do this, the master component must be connected with a web service from a central system, if there is an operation to be carried out, the master component sends the necessary data to the slave component so that it can carry it out. this component has the following subcomponents: a) system controller. it is the base system of the arduino. b) ethernet component. it is an electronic device (ethernet shield) connected to the arduino to be able to communicate with the internet. c) bluetooth component. it is an electronic device to carry out communications with the slave component. d) manage operation. this is the software logic created to be able to read the operations of the master component and send the operations via bluetooth to the slave component. 4○ slave component. this is a custom artifact created to measure the ambient temperature and go to the plant to essentially carry out two actions: measure the state of the soil and apply irrigation. this component operates in the crop moving to the plants with coordinates. 4.2 electronic devices the proposed system has different electronic devices in order to measure data and generate communication with other components. the master and slave components are integrated by the next electronic devices. figure 2 shows the electronic devices for master and slave components. next, the electronic devices in the master and slave components are explained. 1○ master component. arduino uno5 is the base in this component. to bluetooth connection, we used an electronic device hc-05. for ethernet connection, we used an arduino shield. 5https://www.arduino.cc/en/main/arduinoboarduno figure 2: electronic devices in the system paradigmplus (2021) 2:1 https://www.arduino.cc/en/main/arduinoboarduno implementation of an irrigation system to estimate water in hydroponic crops through evapotranspiration 27 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2○ slave component. arduino mega6 is the base in the slave component. in arduino mega there are two sensors connected: humidity fc-28 and temperature dht11. also, like the master component, the slave component has a bluetooth device hc-06. the slave component also includes two motors the type stepper each one is for the x-axis and y-axis. moreover, the slave component has been connected to a servomotor for the z-axis. finally, for irrigation purposes, the slave component has been connected to a solenoid valve that works at 12 volts. 4.3 hydric balance process the system proposed can irrigate each plant. for it, is necessary certain data to calculate the eto and water estimation. figure 3 presents the data necessary for the irrigation and the principal elements are explained as follows. 1○ data from db. it is the parameters configured for a certain crop and stored in a database. the 6https://www.arduino.cc/en/main/arduinoboardmega2560 figure 3: process and data to water stimation paradigmplus (2021) 2:1 https://www.arduino.cc/en/main/arduinoboardmega2560 28 jorge hernandez and karen daza . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . data are about the type of plants, ideal factors (temperature and humidity), specification in distance, and diameter of roots of the plants, among others. 2○ data required measurements sensors db. it is a reading of sensors to get the current temperature and humidity from the slave component. the slave component can read data for a certain plant. 3○ process in the central system. this element is the process to establish the water estimation in the system. the first step is to calculate the eto. next, is necessary to calculate the etc. finally, with the net irrigation and ideal irrigation, the final result is the volume of water for irrigation and the time necessary to on the solenoid valve in order to shoot water in the plant. a) evapotranspiration reference eto: evapotranspiration of reference consists of four inputs and one output, which is the eto. equation 1, which is fao penman-monteith [19], was implemented for time periods. et0 = (0, 48∆(rn − g))+(γ 37t hr+273 u2(e 0(thr)− ea)) ∆ + γ(1 + 0, 34u2) (1) where: eto reference evapotranspiration (mm hour−1) thr air temperature in one hour (○c) g density of heat flow in the soil (mj m−2 hour−1) rn net radiation of the crop surface (mj m−2 hour−1) γ psychometric constant (kpa ○c−1) ∆ slope of the steam precision curve (kpa ○c−1) u2 wind speed at a height (ms−1) e0(thr) pressure of vapor saturation at air temperature (kp a) ea real steam pressure (kp a) the temperatures max min are used to calculate t hr. based on that, we can calculate e0(thr), the saturation pressure (es), and also the steam pressure (es − ea). from this, the average temperature is used to calculate the slope of the steam precision curve (∆). in order to calculate whether data depend on the location of the crop, the information in the database regarding sea level and longitude-latitude coordinates are used. based on this information, we can also calculate the extraterrestrial radiation (ra). finally, the eto is calculated. b) evapotranspiration of the crop under standard conditions etc: it refers to the crop with the following conditions: no environmental factors, good fertilization, and good conditions for crop development. then, the input crop coefficient (kc) is used to relate the eto with the etc, which generates an output that corresponds to etc. c) planting area: it is the third process of water balance, which consists of three inputs and one output. the first input is diameter of roots (r) used to determine the floor area used by each plant. the second input is row distance (dh), which is the distance covered by each row. the third input is distance between plants (dpl), which defines the distance between plants in the row. based on these, the result of the planting area is generated. d) net irrigation: the fourth process consists of five inputs. the first input comes from et c, which in the water balance is the main loss of water. the second input is the last humidity, which is obtained from the prototype before calculating the irrigation. based on this, the calculation of the eto is validated. the third input is irrigation coefficient (kr), which determines the performance, transportation, and failures in the irrigation. the fourth input is paradigmplus (2021) 2:1 implementation of an irrigation system to estimate water in hydroponic crops through evapotranspiration 29 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . runoff coefficient (ke), which is related to the precipitation (p ) of rain that the crop has had as an additional water gain. the last input is the last irrigation, which is the last amount of water applied. e) ideal irrigation: it is the fifth process to perform. it consists of two inputs: the first input is the temperature at the time of irrigation and the second input is ideal factors stored by the plant (humidity, temperature, and amount of water). based on them, we know whether or not the plant is in condition to receive irrigation. f) final result: if it is necessary to water the plant, the final result is the calculation of the time in which the device will irrigate. it is determined by the volume of the hose and according to the flow or quantity of water calculated in the previous steps. 4.4 crop water needs crops need the water retained in the soil for their biological processes. the need for water for each plant increases with the growth of the plant since it varies from the sowing date to the day of harvesting. the need for water can be defined as the volume of water required. the water balance is the process of subtracting the outputs and adding the water inputs. table 2 presents the inputs and outputs of a crop. table 2: factors related to water balance inputs outputs precipitation evapotranspiration amount of water in previous irrigation percolation losses (filtration of water through deep layers of the soil) capillary ascension (the ability of water to ascend through channels) outlet drain 4.4.1 evapotranspiration to crop (etc) in excellent conditions, the crop should be fertilized, free of diseases, and without moisture limitations in the soil. evapotranspiration that occurs in a cultivated area can be measured directly through mass transfer methods using crop evapotranspiration. to calculate etc, we use the following equation: etc = kcet0 (2) where kc is the crop coefficient. kc is determined by various crop factors such as development stage, climate, and plant type. table 3 shows the experimental values used in the software to establish the kc value. the farmer can choose the type of need to kc for a certain crop. 4.4.2 net irrigation (θ) net irrigation is the amount of water suggested from evapotranspiration. to calculate the net irrigation it is necessary to obtain the water requirement of the plant (da). the water requirement of the crop can be expressed in m3/hour and is calculated with the following equation. da = 10 lp ∑ d=1 ((kcetp)− pke100 ))a kr (3) paradigmplus (2021) 2:1 30 jorge hernandez and karen daza . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 3: crop water requirements. type of water need value for (kc) very low 0.3 low 0.5 moderate 0.9 high 1,2 where: the value 10 is the factor that is applied to convert da to m3 / hour. lp is the duration of the growth period. kc ∗ et p is the evapotranspiration of the crop. p is the precipitation. ke is the runoff coefficient. kr is the coefficient of irrigation efficiency. a is the area planted. the proposed net irrigation ( t heta) will be per day (lp = 1). the following equation allows you to calculate the net irrigation. θ = a ∗ 10 ((kcetc)− pke100 ))θi + hsn kr (4) where: θi is net irrigation from the previous day. θ is the crop water requirement m3/hour. hsn is the soil moisture measured in the plant section. 4.4.3 net irrigation using ideal factors per plant. the ideal factors are established by the farmer in the software to establish the water that he would apply at an ideal temperature. to calculate the optimal total amount (rtop) the ideal amount of water (rop), an ideal temperature (top), and the current temperature at the time of irrigation (tins) were used. using the following equation, rtop can be calculated. rtop = roptins top (5) the optimal total amount (rtop) would be the amount that the farmer would apply to irrigation in an ideal case. the optimal total amount is compared to the amount of irrigation that the software calculated ( t heta) using evapotranspiration. taking the variable that is greater, it will compare with the other and it will be added to eliminate estimation errors, then, if t heta <(rtop), the applied water will be: qap = θ + rtop 100 (6) else: qap = rtop + θ 100 (7) 4.4.4 time to irrigation the flow (q) is the volume of water that must pass through a surface in a given time. this section explains how the prototype determines the flow rate to irrigate a certain amount of water. the prototype uses a water flow sensor to allow the water to pass through. to find the flow, the conversion factor (kcon) and the frequency (f) of the pulses with which the rotor of the water flow sensor is driven when the water passes through the mechanism were used. the frequency is calculated by: f = kwithq. to calculate the flow, the following equation was used. paradigmplus (2021) 2:1 implementation of an irrigation system to estimate water in hydroponic crops through evapotranspiration 31 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . q = f kcon (8) to have a more accurate measurement it is necessary to calibrate the value of the conversion factor, this is done by doing experimental tests. it is done by: kcon = f racpulsesv 60. the volume (v) can be calculated as: v = πr2l (9) where: r is the radius of the hose where the water comes out in the prototype. l is the length. to water with a specific amount, it will be done by calculating the time (seconds) in which the flow (water to be watered) needs to pass through the volume of the medium (hose). the calculated time (t) allows you to set the milliseconds with which the prototype should turn on the water flow sensor. to calculate t the following equation was used. t = v qap (10) 4.5 system coordinates the device makes two displacements for calculating the water balance. the first displacement is made to read the humidity of the plant, while the second displacement is made at the time of irrigation. in the crop, each plant has a dropper, where the water is deposited to perform drip irrigation. there are three axes (x, y, z), where axes x, y are used for the displacement ∆x and ∆y respectively. the axis z is used to irrigate. figure 4 presents the cartesian plane implemented in the crop. it illustrates two elements for the axis z: the first one is the electrovalve for the irrigation and the second one is the humidity sensor. figure 4: movements of the prototype in the crop. paradigmplus (2021) 2:1 32 jorge hernandez and karen daza . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 communication between components the client application has direct communication with the central system. master and slave components communicate through bluetooth using a data format created for this project in order to avoid data loss. then, when the master sends information to the slave, the slave produces a response and vice versa; nevertheless, if the master or the slave goes offline, the response is not received; then, the action is sent again and when the offline device goes online they recover the communication to receive the data. the central system responds using json format. then, each component has a handler to generate this format depending on the type of response requested. the fundamental part of the project is in the communication between the master and the slave components. the master component is constantly consulting the rest api as well as the operations that are carried out in the future. table 4 presents the data that the master must send to the slave to perform the operation depending on the type, i.e., humidity reading, temperature reading, go to plant or make irrigation. once data is sent to the slave, it returns the responses of the response slave column. table 4: responses between master and slave communication action (ac) data response from slave temperature ac measurement temperature humidity ac, position (x, y) measurement humidity move to plant ac, position (x, y) ok irrigation ac, position (x, y), time irrigation volumen water the central system handles the http requests in the format json. the central system receives and returns json data. then, each component has a handler to generate this format depending on the type of response requested. the most important handler is the central system, because, receives and sends json responses. figure 5 presents the communication interaction in the central system. in the figure, the central system receives http and bluetooth requests, and depending on the data, the response can be in json or string with data. figure 5: interaction in the handler of central system 4.7 irrigation scheduling this section explains step by step how the system performs the water estimation needed by each of the plants in the crop, with this already defined, it will be possible to program different risks according to the needs according to the data of each plant, in this way when programming the irrigation, the system will carry out this step by step automatically at the time determined by the system administrator. paradigmplus (2021) 2:1 implementation of an irrigation system to estimate water in hydroponic crops through evapotranspiration 33 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . step 1: in this step the management of the plants is done, this means that we validate if there are already plants in our system, otherwise we create the plants that are necessary since we must have a record of all the plants that exist in the crop to make the corresponding measurements for each one. in addition, you can delete plants or edit their data. step 2: in this step the configuration of each plant is recorded, its most relevant requirements, such as the type of plant, its species, and the amount of water it needs, as this may vary depending on the species planted in the crop. step 3: in this step the mechanical device will move to the plant that we indicate from the system and will take the humidity of the plant, this data is sent to the system and is stored for further processing. step 4: in this step, the device takes the temperature of the environment, this data is stored in the system and together with the humidity it is possible to calculate the amount of water that the plant needs. step 5: in this step, the device makes the irrigation of the corresponding amount of water, based on the time obtained in the previous calculations, in this way, at the end of the irrigation time, the irrigation process of the plant is finished. each of the steps mentioned above is necessary for the correct operation of the device, also to obtain the correct calculation for each plant and thus apply the required amount of water. then, the system can be programmed to do the step by step at a certain time every day or when it is required according to the species that are being planted, we call this the irrigation programming or scheduling. 5 results we performed a case study, which corresponds to the implementation of the system in one crop established in bogota, colombia, located at the coordinates: north latitude: 4o35′56′′ and west longitude: 74o04′51′′. temperatures from december to march are high, averaging 20°, but from april to october they are lower, averaging 19° or less. the height at sea level is 2.625 meters. tomato was taken as a study crop. the relative humidity ranges were between 70% and 80%. root length density range was between 0.5 cm and 1.5 cm [20]. the ideal tomato temperature can be divided into three stages: 1○ germination. according to [21], the temperature for germination is between 16 and 30 degrees celsius. 2○ growth development. according to [21], the temperature for growth development is between 12 and 30 degrees celsius. 3○ fructification. the ideal temperature is between 15 and 22 degrees celsius. then, in general, the values between 18.3 and 32.2 degrees celsius were considered ideal air temperatures for tomatoes during the growing season [22]. we tested the system in a crop composed of thirty plants. the distance between rows was 0.7 meters and the distance between plants was 0.4 meters. the crop was observed from day 55 for the fructification stage. we studied ten plants of the thirty plants of the crop. five plants were irrigated using the system, while the other five plants were irrigated manually. during 30 days the system made the water balance for the five selected plants and manual irrigation was carried out for other plants. the irrigation was done every day at 6:00 am by drip establishing different water levels. humidity is measured from 0% to 100%, where 0% means that the soil is completely dry and 100% is completely wet. throughout the process of plant development, humidity must be maintained between 40% and 70% as ideal factors. on the one hand, figure 6 shows the humidity of the five plants irrigated manually, where the highest humidity was 98% achieved by the plants 2 and 3, while the lowest humidity was 0.3% achieved by plant 3. on the other hand, figure 7 shows the humidity of the five plants irrigated by paradigmplus (2021) 2:1 34 jorge hernandez and karen daza . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0 3 6 9 12 15 18 21 24 27 30 0 50 100 days m ea su re d hu m id ity % plant1 plant2 plant3 plant4 plant5 figure 6: humidity measured in plants irrigated manually 0 3 6 9 12 15 18 21 24 27 30 40 50 60 70 days h um id ity % plant1 plant2 plant3 plant4 plant 5 figure 7: humidity measured in plants irrigated with the system the system, where the highest humidity was 71.76% achieved by plant 5, while the lowest humidity was 39.63% also achieved by the plant 5. with this in mind, we can observe that the humidity levels for the plants irrigated by the system are in almost all cases between the ideal factors i.e., between 40% and 70%, while the humidity levels for the plants irrigated manually are out of the ideal factors. thus, based on the experimentation and the corresponding results, the system ensures a better irrigation process and also saves water in the crop. in addition, figure 8 presents the average (thr), minimum (tmin) and maximum (tmax) tem0 5 10 15 20 25 30 15 20 25 30 days te m pe ra tu re ○ c. thr tmax tmin figure 8: experimental temperature readings. paradigmplus (2021) 2:1 implementation of an irrigation system to estimate water in hydroponic crops through evapotranspiration 35 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0 5 10 15 20 25 30 0 2 4 days et o (m m /h ) eto1 eto2 figure 9: experimental data eto. perature per day for this period of time. these readings were obtained by the sensor installed in the prototype. thus, the temperature oscillates between 32.00 and 16.00 degrees celsius. figure 9 presents the eto readings in two time ranges: 06.00 07.00 (eto1) and 14.00 15.00 (eto2) during the thirty days. these results were dependent on thr according to the min and max temperature readings 24 hours back. then, in the crop the eto2 was higher in the days 1 to 13, 16 to 21 and 28 to 30. for eto1 it oscillated between 0.2 to 4.0 (mm/hour) and for eto2 it had a variation between 0.9 to 4.6 (mm/hour). 5.1 information system the client application is the system that the user can be interacting with. we named the client application riwa. this is the system that we provide to the farmers. mainly, the farmer can easily register any plant in your crop. for the plant is necessary to set the name, name species, diameter or roots, and its ideal factors (humidity and temperature). in riwa, the user can register plants, view metrics about the water irrigation, establish crops variables, schedule the irrigations and make irrigation (right away). figure 10 shows an interface of the riwa system to present the water consumption of the irrigation by months. 6 discussion and future work 1○ crops implementation. there is a huge gap between technology and agriculture in underdeveloped countries. since the implementation of this would make the crops even more expensive, in addition to that, farmers refuse to try to change their way of producing, but we believe that everything is a process. with the use of technology in the crops will be even easier for the promotion of urban agriculture. thus, farmers will be able to opt for change and implement technology, with this they will be able to have greater control of the crop remotely, which will not require the investment of large water resources. 2○ robot to go plant to plant. we show that the approach allows performing the function of estimating and watering the plants, solving the problem of calculating a good use of water. we try to make the software system very practical and intuitive so that farmers can interact without any problem. at first, it was thought to build a robot to go plant to plant, due to cost and energy issues using the robot, this idea did not give results. to solve cost issues, the prototype is transported plant to plant by means of metal guides lowering to the floor of the plant to take the sample. the proposed approach allows going plant to plant with the limitation that if the plant has a high paradigmplus (2021) 2:1 36 jorge hernandez and karen daza . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 10: irrigation report proportion, the metal guides must be changed since the sensor would not reach the ground. this approach saves the purchase of sensors per plant. 3○ water analytic. this point presents the future work. water quality is a not factor used in this approach. water quality refers to the characteristics of a water supply that will influence its suitability for a specific use, i.e. how well the quality meets the needs of the user [23]. the idea is to establish the water quality in each crop after a month’s use system. thus, the farmer can know if the water is good or not through the water quality index (wqi). the idea is to create a new artifact to measure the required attributes (phosphate, nitrates, etc. ) to establish the wqi. the new artifact must be integrated with the main system of this approach. 7 conclusions with the system, together with each of the electronic devices and software components explained, the agrarians can have a significant tool to irrigate their crops by making a water estimation to make a precise water use. in addition, the system explained favors the work of the farmers, since they do not have to irrigate every day because the system enables programming the irrigation for the plants making the validation of whether to irrigate or not by calculating the water needed for each plant. in the system, the farmers can adjust certain parameters of soil and factors of the plants in order to control the water estimation. the evapotranspiration was calculated in each water estimation with hour time intervals. in that order of ideas, the eto range was [0.2 4.6] mm/h in the studied crop. the paradigmplus (2021) 2:1 implementation of an irrigation system to estimate water in hydroponic crops through evapotranspiration 37 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . humidity range was [40% 70%] (optimal). the water estimation was 2% for the crop because the general soil and plants were equal in the studied time of the crop. eto estimations establish that irrigation for the crop location and the species studied is best done in the morning when temperatures are low, this generates a lower water use because water loss by evapotranspiration eto is lower. water is a very important factor in life. most of the processes use the water to generate something, therefore, it is pertinent to interact technologically to contribute to saving water. the ideal factors of a plant (humidity, temperature) do not determine the amount of water to irrigate, in addition to these, the soil intervenes, which is the one that retains the water. to really calculate the water you have to measure the soil factors. authors’ information – jorge hernandez is a master’s student of information and communication sciences at the universidad distrital francisco josé de caldas. his research topics are internet of things, artificial neuronal networks, dynamic behavior, and machine learning – karen daza is an undergrad student of telematic engineering at the universidad distrital francisco jose de caldas. authors’ contributions – jorge hernandez designed and implemented the algorithm for the operation of the device, worked on the generation of the results, and participated in the writing of the article. – karen daza conducted a literature review of the main components of the device, collaborated in the analysis of the results, and participated in the writing of the article. competing interests the authors declare that they have no competing interests. funding no funding was received for this project. references [1] b. k. joe stevens, “trends and outlook: agricultural water management in southern africa country report south africa,” international water management institute, pp. 10–18, 2015. [2] c. j. van der kooi, m. reich, m. löw, l. j. de kok, and m. tausz, “growth and yield stimulation under elevated co2 and drought: a meta-analysis on crops,” environmental and experimental botany, vol. 122, pp. 150–157, 2016. [3] s. nonhebel, “the effects of use of average instead of daily weather data in crop growth simulation models,” agricultural systems, vol. 44, no. 4, pp. 377–396, 1994. [4] q. zheng, f. zhu, c. mao, and x. lv, “a smart irrigation decision support system based on cloud,” undergraduate, hohai university, nanjing, china, pp. 1–10, 2014. paradigmplus (2021) 2:1 38 jorge hernandez and karen daza . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [5] k. daza, j. hernandez, and h. florez, “hardware and software system for hydric estimation and crop irrigation scheduling,” in international conference on computational science and its applications, pp. 150–165, springer, 2019. [6] p. bradley and c. marulanda, “simplified hydroponics to reduce global hunger,” in world congress on soilless culture: agriculture in the coming millennium 554, pp. 289–296, 2000. [7] c. camp, “subsurface drip irrigation: a review,” transactions of the asae, vol. 41, no. 5, p. 1353, 1998. [8] r. allen, l. pereira, d. raes, and m. smith, “guidelines for computing crop water requirements,” geophysics, vol. 156, p. 178, 1998. [9] s. irmak and d. z. haman, “evapotranspiration: potential or reference,” ifas extension, abe, vol. 343, 2003. [10] j. ritchie, “soil water balance and plant water stress,” in understanding options for agricultural production, pp. 41–54, springer, 1998. [11] g. hoogenboom, j. jones, p. wilkens, c. porter, w. batchelor, l. hunt, k. boote, u. singh, o. uryasev, w. bowen, et al., “decision support system for agrotechnology transfer version 4.0,” university of hawaii, honolulu, hi (cd-rom), 2004. [12] m. gocic and s. trajkovic, “software for estimating reference evapotranspiration using limited weather data,” computers and electronics in agriculture, vol. 71, no. 2, pp. 158–162, 2010. [13] s. zhang, m. wang, w. shi, and w. zheng, “construction of intelligent water saving irrigation control system based on water balance,” ifac-papersonline, vol. 51, no. 17, pp. 466–471, 2018. [14] n. shah and i. das, “precision irrigation sensor network based irrigation,” problems, perspectives and challenges of agricultural water management; intech: rijeka, croatia, pp. 217–232, 2012. [15] s. m. umair and r. usman, “automation of irrigation system using ann based controller,” international journal of electrical & computer sciences ijecs-ijens, vol. 10, no. 02, pp. 41–47, 2010. [16] s. shamshirband, m. amirmojahedi, m. gocić, s. akib, d. petković, j. piri, and s. trajkovic, “estimation of reference evapotranspiration using neural networks and cuckoo search algorithm,” journal of irrigation and drainage engineering, vol. 142, no. 2, p. 04015044, 2016. [17] g. suciu, t. us, urelu, c. m. bălăceanu, and m. anwar, “adaptation of irrigation systems to current climate changes,” in international conference on business information systems, pp. 534–549, springer, 2018. [18] l. prisilla, p. rooban, and l. arockiam, “a novel method for water irrigation system for paddy fields using ann,” tech. rep., ijcsn, 2012. [19] l. zotarelli, m. d. dukes, c. c. romero, k. w. migliaccio, and k. t. morgan, “step by step calculation of the penman-monteith evapotranspiration (fao-56 method),” institute of food and agricultural sciences. university of florida, 2010. [20] r. m. machado and g. o. maria do rosàrio, “tomato root distribution, yield and fruit quality under different subsurface drip irrigation regimes and depths,” irrigation science, vol. 24, no. 1, pp. 15–24, 2005. [21] s. adams, k. cockshull, and c. cave, “effect of temperature on the growth and development of tomato fruits,” annals of botany, vol. 88, no. 5, pp. 869–877, 2001. paradigmplus (2021) 2:1 implementation of an irrigation system to estimate water in hydroponic crops through evapotranspiration 39 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [22] c. everhart, “the complete guide to growing tomatoes: a complete step-by-step guide including heirloom tomatoes (back-to-basics gardening),” 2010. [23] r. s. ayers, d. w. westcot, et al., water quality for agriculture, vol. 29. food and agriculture organization of the united nations rome, 1985. paradigmplus (2021) 2:1 introduction hydroponic crops evapotranspiration (et) hydric estimation related work proposed approach components of system electronic devices hydric balance process crop water needs evapotranspiration to crop (etc) net irrigation () net irrigation using ideal factors per plant. time to irrigation system coordinates communication between components irrigation scheduling results information system discussion and future work conclusions vol. 1, no. 3, pp 23 43, 2020 doi: 10.55969/paradigmplus.v1n3a2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . assessing employee satisfaction in the context of covid-19 pandemic ana fernandes id 1, rui lima id 2, margarida figueiredo id 1, jorge ribeiro id 3, josé neves id 4, and henrique vicente id 1,4,� 1universidade de évora, évora, portugal anavilafernandes@gmail.com,{mtf,hvicente}@uevora.pt 2instituto politécnico de saúde do norte, famalicão, portugal rui.lima@ipsn.cespu.pt 3instituto politécnico de viana do castelo, viana do castelo, portugal jribeiro@estg.ipvc.pt 4universidade do minho, braga, portugal jneves@di.uminho.pt abstract the current covid-19 pandemic crisis brought new challenges for all companies, forcing them to adopt new working methods to avert/minimize infection. monitoring employee satisfaction is a challenging task, but one that is paramount in the current pandemic crisis. a workable problemsolving methodology has been developed and tested to respond to this challenge that examined the dynamics between artificial intelligence, logic programming, and entropy for knowledge representation and reasoning. such formalism are in line with an artificial neural network approach to computing. the ultimate goal is to assess employees’ satisfaction in water analysis laboratories while considering its development and management. the model was trained and tested with realworld data collected through questionnaires. the proposed supervised exercise yielded an overall accuracy of 92.1% and 90.5% for both, training and testing sets. keywords: covid–19 ⋅ human resources management ⋅ organizational performance ⋅ artificial intelligence ⋅ logic programming ⋅ entropy ⋅ knowledge representation and reasoning ⋅ artificial neural networks received: 21 november 2020 ⋅ accepted: 19 december 2020 ⋅ published: 30 december 2020. 1 introduction the recent outbreak of the coronavirus (covid-19) is an unprecedented situation worldwide. the world health organization (who) and public health authorities around the world are taking steps to slow the outbreak of covid-19 and reduce infection rates. while many companies are developing and implementing business continuity plans, all employers need to prepare, get along and respond appropriately as the covid-19 pandemic evolves [1]. the covid-19 spreads not only through respiratory droplets or contact with contaminated surfaces, but through [2], viz airborne particles and fomites. this article is distributed under the terms of the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. https://doi.org/10.55969/paradigmplus.v1n3a2 https://orcid.org/0000-0002-1358-1372 https://orcid.org/0000-0003-1107-3785 https://orcid.org/0000-0002-8969-0380 https://orcid.org/0000-0003-1874-7340 https://orcid.org/0000-0002-8863-0351 https://orcid.org/0000-0001-8456-7773 24 ana fernandes et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . the risk of exposure to covid-19 in the workplace depends on the likelihood of reaching one meter from other people, frequent physical contact with people who may be infected with covid-19 and contact with contaminated surfaces and objects [3]. the who identified three levels of exposure risk in workplace [4], viz: – low exposure risk – jobs or work without frequent, close contact with the general public or others. workers in this group have minimal occupational contact with the public and other co-workers. examples of such chores may include remote workers (i.e. working from home), office workers without frequent close contact with others and workers providing teleservices. – medium exposure risk – activities or assignments with close, frequent contact with the general public or others. this risk level may apply to workers who have frequent and close contact with the people in high-population-density work environments (e.g. food markets, bus stations, public transport, and other work activities where physical distancing of at least one meter may be difficult to observe), or tasks that require close and frequent contact between co-workers. this may also include frequent contact with people returning from areas with community transmission. examples of such duties may include front-line workers in retail, home deliveries, accommodation, construction, police and security, public transport, and water and sanitation. workplaces for medium-risk jobs require daily cleaning and disinfection at least twice a day of objects and surfaces that are touched regularly, including all shared rooms, surfaces, floors, bathrooms and changing rooms [5]. – high exposure risk – jobs or tasks with close contact with people who may be more likely to have covid-19, as well as contact with objects and surfaces possibly contaminated with the virus. examples include transporting people known or suspected to have covid-19 without separation between the driver and the passenger, providing domestic services or home care for people with covid-19, and having contact with the deceased who were known or suspected of having covid-19 at the time of their death. employments that may fall under this category include domestic workers, social care workers, personal transport and home delivery providers and repair technicians (e.g., plumbers, electricians) who have to provide services in the homes of people with covid-19. high-risk workplaces require specific measures since the possibility of suspending activity must be assessed. requires cleaning and disinfection before and after contact with any known or suspected case of covid-19, use of a medical mask, disposable lab coat, gloves and eye protection for workers who come into contact with suspected or known people with covid-19 [5]. aside from the above, there may be jobs in the same work environment with different levels of risk and different jobs or work items may have similar exposure levels. thus, managers must carry out risk assessments to determine the possibility of exposure risk, to implement preventive measures [4]. the risk assessment must be carried out for each specific work environment and to each allocation. following this line of thinking, for each risk assessment it is prudent to consider the environment, the task, the threat, if any (e.g., for front-line personnel) and available resources, such as personal protective equipment. furthermore, the specificities of each worker should be considered since some workers may be at a higher risk of developing serious covid-19 disease due to age or pre-existing medical conditions [5]. however, the implementation of the action plan and the preventive measures is not sufficient. all new processes and working measures must be understood and accepted by each one to guarantee the commitment of the entire team [6]. however, assessing employee satisfaction is a difficult task that comprises a variety of issues that depend on both the employee and the organization [7, 8]. the present work aims to assess the satisfaction of the employees of water analysis laboratories (sector classified by who as to have a medium risk of exposure to the virus [4]) with the preventive measures resulting from the pandemic crisis. paradigmplus (2020) 1:3 assessing employee satisfaction in the context of covid-19 pandemic 25 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 related work in the literature, employee satisfaction studies are common, but there are still not many studies related to employee satisfaction in times of a pandemic because it is a recent topic. using the sciencedirect platform the search for the term covid-19 gave 30.698 results. however, combining the terms covid-19, risks and employee satisfaction only 256 results were obtained. studies related to covid19 and employee satisfaction in the areas of health [9, 10] and in the hospitality sector [11-13] are the most frequent. 2.1 health risks tengilimoglu et al. studied the levels of anxiety, depression, and stress of health employees during the covid-19 pandemic in turkey. the authors stated that the biggest cause of anxiety or stress among health professionals comes from the fear of contaminating their family members with the covid19 virus [9]. zhang et al. predicted the anxiety and satisfaction of health professionals during the covid-19 pandemic based on a cohort composed by 712 healthcare workers [10]. grounded on an 8-point framework the authors present an empirical measure of covid-19 organizational support and investigate its associations with healthcare workers’ anxiety and life satisfaction. according to the authors the proposed measure enables healthcare institutions to assess, monitor, compare and improve covid-19 specific support and contribute to minimize the anxiety levels of the collaborators during the pandemic [10]. 2.2 hospitality industry with regard to the hospitality industry, bajrami et al. examined various effects of covid-19 (e.g., job insecurity, complaints from employees during isolation, risk behavior in the workplace and changes in the organization) on work motivation, job satisfaction and sales intentions [11]. the study was carried out on serbia and embraced a cohort of 624 hospitality workers. the authors point out that the job insecurity and changes in the organization showed a negative impact in all outcomes, whereas while risk-taking behavior has a negative influence only in job satisfaction. regarding demographic variables only the age and marital status influence job motivation and turnover intentions [11]. farmaki and stergiou explored front-line hotel employees’ views about working during the covid-19 pandemic to identify the factors that may influence their ability and willingness to report to work. despite the difficulties in the distinction between barriers to ability or willingness to report to work the authors proposed some remedial recommendations such as training, measures to facilitate employees’ transport, measures for infection controls in customer areas, and the reinforce the communication between the organization and the employees [12]. jung et al. examined the effects of job insecurity caused by covid-19 on job engagement and turnover intent based on a cohort comprising 314 employees of seoul five-star hotels. according the authors the perceptions of job insecurity had negative effects on the engagement of deluxe hotel employees and led to the increased of turnover intent [13]. the study also reveals that the job insecurity had a greater influence on employees’ engagement; e.g., if one state that generation y than generation x, one is asserting that the impact of job insecurity is higher for people of generation y than for people of generation x [13]. kang and shin examined how risk reduction strategies using technology innovation for social distancing and cleanliness (e.g., kiosk check-in system, mobile check-in system, robot cleaning system, and ultraviolet light cleaning system) at hotels influence customer health risk perception and decision-making behaviors in both pandemic and post-pandemic scenarios. the study allowed to identify that the perceived health risk mediates the relationship between expected interaction and hotel booking intention. the authors highlighted that low levels of expected interaction through technology mediated systems lead to low levels of perceived health risk [14]. paradigmplus (2020) 1:3 26 ana fernandes et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 fundamentals 3.1 logic programming there are many approaches to knowledge representation and reasoning (krr) using the epitome of logic programming (lp), namely in the areas of model theory and proof theory. in this article, the proof theoretical methodology for problem solving was adopted and expressed as an extension of the lp language [15]. under this setting a lp will be grounded on a finite set of clauses as presented in listing 1. listing 1: the prime example of a logic program 1 { 2 ¬p ← not p, not exceptionp 3 p ← p1 , .. , pn , not q1 , .. , not qn 4 ?(p1 , .. , pn , not q1 , .. , not qn )n, m ≥ 0 5 exceptionp1 , .. , exceptionpj (0 ≥ j ≥ k) , k ∈ z 6 } the first clause denotes predicate’s closure, "," designates "logical and", while "?" is a domain atom denoting "falsity", the pi, qj, and p are classical ground literals, i.e., either positive atoms or atoms preceded by the classical negation sign ⇁ [15]. indeed, ⇁ stands for strong negation, i.e., a strong declaration that speaks for itself, while not denotes negation-by-failure, i.e., a failure in proving a certain statement since it was not declared in an explicit way. according to this way of thinking, a set of exceptions (or abducibles) are present in every program, given here in the form of exemptions to the extensions of the predicates that make the program, i.e., clauses of the form exceptionp1 , .. , exceptionpj (0 ≥ j ≥ k), k ∈ z that denote data, information or knowledge that cannot be ruled out [16]. on the other hand, clauses of the type ?(p1, .. , pn, not q1, .., not qn)n, m ≥ 0 are integrity constraints (ics) (or invariants) that make it possible to specify the context under which the universe of discourse should be understood [15, 17]. it is undeniable that expressing theories as logic programs has become more natural and general as the field of lp matured and other fields began to use its tools and results. theories are typically uttered as a series of patterns (rules) and facts that make it possible to infer non-logical consequences using logical inference. in writing such rules and facts, both explicit (or strong) negation and explicit declarations may be used to reinforce knowledge. theories, on the other hand, can be further refined by adding special rules in the form of ics. these assume that, regardless of the assumptions, some conditions must be met. an implicit constraint on any sound theory has overall consistency, i.e., it must not be possible to infer a conclusion and its contradiction. since the most common situation in the real world is incomplete and updateable information, any system that makes serious attempts to deal with real-world situations must cope with such complexities. the principle used is the open world assumption (owa) where everything is unknown or undefined until one has some secure evidence of its truthfulness or falseness. indeed, this principle differs from the more usual closed world assumption (cwa) where everything is assumed false until there is solid evidence of its truthfulness. it was not taken a fuzzy logic approach once thresholds would be needed. for such attitude an a priori evaluation of falseness or truthfulness would be required. 3.2 a thermodynamics approach to knowledge representation and reasoning energy transfer operations or processes are systematically analyzed to determine how they can be assessed. it turns out that energy transfer should not only be characterized by the act or process itself but should also be seen in a broader context. this context is introduced as the entropic potential of paradigmplus (2020) 1:3 assessing employee satisfaction in the context of covid-19 pandemic 27 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . the transferred energy. it takes into account the entire energy transfer in its initial and finite states, i.e., it starts, for example, as pure exergy if it is a primary energy, and ends as pure anergy if it has become part of the internal energy of the environment. with this in mind, an energy degradation process can be defined under a proper tangible background. indeed, the problem-solving methodology presented and discussed in this article is based on this perception and aims to describe the practices of krr as a process of energy degradation [17, 18, 19]. in order to explain the basic rules of the proposed approach, the first and second law of thermodynamics are considered, attending that one’s system moves from state to state over time. the first part, also known as the energy saving law, states that the total energy of an isolated system is constant, i.e., cannot change over time. this means that energy can be converted but cannot be generated or destroyed. the second part deals with entropy, a property that quantifies the orderly state of a system and its evolution. these characteristics fit the proposed vision of krr practices, and have to be understood as a process of energy degradation or, in other words, when one looks to system entropic state, its energy can be break up and used in sense of erosion, but never used in the sense of destruction, viz: – exergy, sometimes called available energy or more precisely available work, is the part of the energy which can be arbitrarily used after a transfer operation or, in other words, the entropic state of the universe of discourse. in figure 2 it is given by the dark colored areas. – vagueness, i.e., the corresponding energy values that may or may not have been transferred and consumed. in figure 2 are given by the gray colored areas. – anergy, that stands for an energetic potential that was not yet transferred and consumed, being therefore available, i.e., all of energy that is not exergy. these terms refer to all possible processes as pure energy transfer and consumption practices [17, 18, 19]. 3.3 artificial neural networks artificial neural networks (anns) are computing tools inspired by studies of the human brain and nervous system. in fact, they are mathematical models that simulate such systems as are understood today. one of the main characteristics of anns is their ability to learn. in fact, it is important to note that anns are not traditional computer programs. they learn from examples through a process called training, in which anns organize themselves to adjust an internal set of parameters (i.e. synapses weights) that are used to collect the information contained in the data. compared to traditional methods, anns treat inaccurate and/or incomplete data, give approximate results, and are less prone to outliers. in addition, it is not necessary to accept restrictions or to know the relationships between variables from the outset. multi-layer perceptron (mlp) is one of the most widespread ann architectures in which neurons are layered and only forward connections exist [20]. the mlp design is typically trial and error using an upward approach, starting with an initial architecture that is adjusted to minimize the internal error (e.g., mean square error) [20, 21]. 4 material and methods 4.1 employees the study was carried out in a water laboratory in southern portugal. the age of the employees varied between 18 and 60 years (mean age 38 ± 20 years), with 61% women and 39% men. a questionnaire to assess employee satisfaction with the new pandemic crisis procedures was created and used for a cohort of 64 elements. the questionnaire was used over a period of five months (june to october) and resulted in a database of 320 records. in order to avoid possible hidden errors in connection with sampling procedures, the questionnaires were applied to all employees of the laboratories [22]. paradigmplus (2020) 1:3 28 ana fernandes et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 questionnaire the questionnaire was divided into two segments; the former one contained frequently asked questions (e.g. age, gender, educational qualification, seniority and department within the organization), while the latter contained points for assessing employee satisfaction with the current protective measures. in the first part of the questionnaire the answers are descriptive, while in the others the likert scale was used with four levels, i.e., very dissatisfied, slightly dissatisfied, satisfied and very satisfied. in addition, the participants are requested to specify the tendency of his/her answer, i.e., growing tendency (very dissatisfied → very satisfied) or the opposite (very satisfied → very dissatisfied). 4.3 methodology the action research methodology for problem solving, which is also known as participatory action research, community-based study, co-operative enquiry, action science and action learning – is an approach commonly used for improving conditions and practices in a range of environments and will be used in this work. it involves people conducting systematic enquiries in order to help them improve their own practices, which in turn can enhance their working environment and the working environments of those who are part of it. the purpose of undertaking action research is to bring about change in specific contexts. action research’s strength lies in its focus on generating solutions to practical problems and its ability to empower people, by getting them to engage with research and the subsequent development or implementation activities. 5 results and discussion 5.1 sample characterization the gathered data was classified by age-groups as shown in figure 1(a) (i.e., 17-20, 21-30, 31-50 and 51-70 years old). 69% of the partakers are younger than 30 years of age while 6% are older than 51 years of age. in terms of academic qualifications, 80% of the cohort reported having basic or secondary education, 17% reported having a degree and 3% reported having postgraduate education (figure 1(b)). in terms of length of service, 50% of participants reported 1-5 years, 32% more than 11 years, and 12% less than 1 year (figure 1(c)). in terms of departmental areas (figure 1(d)), 34% belong to the management sector (consisting of quality management, human resources, finance and results departments), and 66% belong to the technical sector (i.e. chemistry, microbiology and sampling departments)). 5.2 a thermodynamic approach to data attainment and processing aiming to collect information about employees’ satisfaction concerning the new procedures related with the pandemic crisis, the second part of the questionnaire comprises questions related with the employees’ opinions about organization related items – four items (ori – 4), training related items – four items (tri – 4), resources related items – four items (rri – 4), and cleaning and disinfection related items – five items (cdri – 5). since the participants were requested to mark the alternative(s) that best reflect their feelings concerning each item and the tendency of his/her answer, i.e., growing tendency (very dissatisfied → very satisfied) or the opposite (very satisfied → very dissatisfied), the answer options were given in terms of an expanded version of a likert scale, viz very satisfied (4), satisfied (3), dissatisfied (2), very dissatisfied (1), dissatisfied (2), satisfied (3), very satisfied (4). in order to quantify the qualitative information and make the process intelligible, complete calculation details for organization related items – four items (ori – 4) are provided. table 1 shows the results regarding an employee answers to the ori – 4. for example, the answer to i1 was dissatisfied paradigmplus (2020) 1:3 assessing employee satisfaction in the context of covid-19 pandemic 29 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 1: sample characterization in terms of age groups (a), academic qualifications (b) length of service (c), and (d) departmental areas (2) → satisfied (3), shows a trend in the employee’s opinion with a decrease in entropy, i.e., there is an increase trend in his/her opinion. for i2 and i4 the answers were dissatisfied (2) and satisfied (3), respectively, a fact that speaks for itself, while for i3 no options were pointed out, which indicates a vague situation, i.e., the value of the energy consumed is unknown, although it is known that it is in the bandwidth is the interval 0...1. table 1: an employee’s answers to ori – 4 items scale(4) (3) (2) (1) (2) (3) (4) vagueness i1 × × i2 × i3 × i4 × figure 2 and figure 3 describe such responses regarding the different forms of energy, i.e., exergy, vagueness and anergy. bearing in mind the fact that the markings on the axis correspond to one of the possible scaling options, the employees’ opinion behaves better when the entropy decreases, which is the case with i1, as shown in table 2 for the best and worst case scenarios (bcs/wcs). table 2: evaluation of the entropic state of the system for the best and worst-case scenarios, attending an employee perception in terms of his/her answer to the ars – 4 items best-case scenario (bcs) worst-case scenario (wcs) paradigmplus (2020) 1:3 30 ana fernandes et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i1 i2 i3 i4 paradigmplus (2020) 1:3 assessing employee satisfaction in the context of covid-19 pandemic 31 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 2: a global view of the energy that an employee is predicting to be used by the system to be in the entropic state suggested by his/her responses to ori – 4, per item. the dark areas stand for exergy, gray areas denote vagueness and white areas represent anergy the formula used in table 2 report the way to handle sub-sections of a circle area, they are selfexplanatory. on the other hand, the data collected above may now be structured in terms of the predicate organization related items (ori – 4) in the form ori − 4 ∶ exergy, v agueness, an ergy, employee satisf action assessment, quality −of −inf ormation → t rue, f alse whose extent follows (table 3 and listings 2 and 3). table 3: the extent of the ori – 4’s predicate as a result of an employee answers to ori – 4 items exbcs va bcs an bcs esa bcs qoi bcs ex wcs va wcs an wcs esa wcs qoi wcs ori – 4 0.31 0 0.69 0.95 0.69 0.31 0.33 0.36 0.77 0.36 paradigmplus (2020) 1:3 32 ana fernandes et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 3: a global view of the energy an employee predicts that has been used by the system to be in the entropic state his/her answer suggests in answering ori – 4 for best (a) and worst (b) case scenarios, per energy type. the dark areas stand for exergy, gray areas denote vagueness and dashed areas represent anergy listing 2: the extent of the predicate ori – 4 for the best-case scenario 1 { 2 ¬ ori − 4(ex, v a, an, esa, qoi) 3 ← not ori − 4(ex, v a, an, esa, qoi), 4 not exceptionori−4(ex, v a, an, esa, qoi) 5 ori − 4(0.31, 0, 0.69, 0.64, 0.69) 6 } listing 3: the extent of the predicate ori – 4 for the worst-case scenario 1 { 2 ¬ ori − 4(ex, v a, an, esa, qoi) 3 ← not ori − 4(ex, v a, an, esa, qoi), 4 not exceptionori−4(ex, v a, an, esa, qoi) 5 ori − 4(0.31, 0.33, 0.36, 0.77, 0.36) 6 } the evaluation of employees satisfaction assessment (esa) and quality of information (qoi) for the different items that make the ori – 4 are now given in the form, viz: paradigmplus (2020) 1:3 assessing employee satisfaction in the context of covid-19 pandemic 33 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . – esa is figured out using esa = √ 1 − es2 (see figure 4), where es stands for the exergy’s that may have been consumed, a value that ranges in the interval 0...1. in the best-case scenario, es = exergy, while in the worst-case scenario, es = exergy + vagueness. esabcs = √ 1 −(0.31)2 = 0.95 esaw cs = √ 1 −(0.31 + 0.33)2 = 0.77 figure 4: esa evaluation – qoi is evaluated in the form, viz. qoi = 1 − es/interval length(= 1) qoibcs = 1 − 0.31 = 0.69 qoiw cs = 1 −(0.31 + 0.33) = 0.36 to complement table 1, table 4 and table 5 shows an employee answers to tri – 4, rri – 4, and cdri – 5. table 4: answers from an employee to the tri – 4, rri – 4, and cdri – 5 items scale(4) (3) (2) (1) (2) (3) (4) vagueness tri – 4 s5 × × s6 × s7 × s8 × rri – 4 s9 × s10 × × s11 × s12 × cdri – 5 s13 × s14 × s15 × s16 × s17 × in table 5, the row global corresponds to the average of the item (e.g., in the column ex bcs, 0.40 = (0.31 + 0.53 + 0.34 + 0.41)/4). paradigmplus (2020) 1:3 34 ana fernandes et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 5: the ranges of the predicates ori – 4, tri – 4, rri – 4, and cdri – 5, which were determined according to an employee’s answers to the ori – 4, tri 4 – 4, rri – 4, and cdri – 5 items exbcs va bcs an bcs esa bcs qoi bcs ex wcs va wcs an wcs esa wcs qoi wcs ori – 4 0.31 0 0.69 0.95 0.69 0.31 0.33 0.36 0.77 0.36 tri – 4 0.53 0 0.47 0.85 0.47 0.53 0.36 0.11 0.46 0.11 rri – 4 0.34 0 0.66 0.94 0.66 0.34 0.36 0.3 0.71 0.3 cdri – 5 0.41 0 0.59 0.91 0.59 0.41 0 0.59 0.91 0.59 global 0.40 0 0.6 0.91 0.6 0.4 0.26 0.34 0.71 0.34 5.3 the esa implementation – a logic programming line 5.3.1 the formal framework logic programming (lp) is a programming paradigm which is largely based on pure logic. any program written in a logic programming language is a set of sentences in logical form, expressing facts and rules about some problem domain, i.e., a problem-solving methodology that involve expressing problems and their solutions in a way that a computer can execute using logic inference. indeed, lp is a creative process that uses deductive methods for problem solving based on the presumption that an attempt is made to construct evidence for the presumption. in this subsection a mathematical logical program is presented that considers the perception of the employee with respect to a particular subject (individual opinions about the satisfaction with the safety procedures) in order to evaluated the organization as a whole (programs 4 and 5). this framework provides the basis for a symbolic assessment of the level of employees’ satisfaction of an organization, plus a measure of its sustainability (qoi), i.e., a set of truth values that range in the interval 0 ... 1 [17, 23]. 5.3.2 the best-case scenario regarding the best-case scenario one may have the program presented in listing 4. the extensions of predicates ori-4, tri-4, rri-4, and cdri-5 are now put together. it is now possible to generate the data sets that will allow one to train an ann for the best-case scenario (see figure 5) [17, 23], viz. – the input in the form of the employees’ opinions about organization related items – four items (extent of predicate ori-4), training related items – four items (extent of predicate tri-4), resources related items – four items (extent of predicate rri-4), and cleaning and disinfection related items – five items (extent of predicate cdri-5). – the output in terms of an evaluation of the employee satisfaction assessment (esa) of an institution and its sustainability (qoi), truth values that range in the interval 0 ... 1. listing 4: the make-up of the lp that establishes the esa implementation in the best-case scenario 1 { 2 /* the s e n t e n c e b e l o w s t a t e s that the e x t e n t of p r e d i c a t e ori -4 is ⤦ ç b a s e d on the e x p l i c i t l y s p e c i f i e d c l a u s e s and t h o s e that ⤦ ç c a n n o t be d r o p p e d */ 3 ¬ ori − 4(ex, v a, an, esa, qoi) 4 ← not ori − 4(ex, v a, an, esa, qoi), 5 not exceptionori−4(ex, v a, an, esa, qoi) 6 7 /* the f o l l o w i n g s e n t e n c e s t a n d s for an a x i o m of ori -4 */ paradigmplus (2020) 1:3 assessing employee satisfaction in the context of covid-19 pandemic 35 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 ori − 4(0.31, 0, 0.69, 0.95, 0.69) 9 10 /* the s e n t e n c e b e l o w s t a t e s that the e x t e n t of p r e d i c a t e tri -4 is ⤦ ç b a s e d on the e x p l i c i t l y s p e c i f i e d c l a u s e s and t h o s e that ⤦ ç c a n n o t be d r o p p e d */ 11 ¬ tri − 4(ex, v a, an, esa, qoi) 12 ← not tri − 4(ex, v a, an, esa, qoi), 13 not exceptiontri−4(ex, v a, an, esa, qoi) 14 15 /* the f o l l o w i n g s e n t e n c e s t a n d s for an a x i o m of tri -4 */ 16 tri − 4(0.53, 0, 0.47, 0.85, 0.47) 17 18 /* the s e n t e n c e b e l o w s t a t e s that the e x t e n t of p r e d i c a t e rri -4 is ⤦ ç b a s e d on the e x p l i c i t l y s p e c i f i e d c l a u s e s and t h o s e that ⤦ ç c a n n o t be d r o p p e d */ 19 ¬ rri − 4(ex, v a, an, esa, qoi) 20 ← not rri − 4(ex, v a, an, esa, qoi), 21 not exceptionrri−4(ex, v a, an, esa, qoi) 22 23 /* the f o l l o w i n g s e n t e n c e s t a n d s for an a x i o m of rri -4*/ 24 rri − 4(0.27, 0, 0.73, 0.96, 0.73) 25 26 /* the s e n t e n c e b e l o w s t a t e s that the e x t e n t of p r e d i c a t e cdri -5 is ⤦ ç b a s e d on the e x p l i c i t l y s p e c i f i e d c l a u s e s and t h o s e that ⤦ ç c a n n o t be d r o p p e d */ 27 ¬ cdri − 5(ex, v a, an, esa, qoi) 28 ← not cdri − 5(ex, v a, an, esa, qoi), 29 not exceptioncdri−5(ex, v a, an, esa, qoi) 30 31 /* the f o l l o w i n g s e n t e n c e s t a n d s for an a x i o m of cdri -5*/ 32 rri − 4(0.41, 0, 0.59, 0.91, 0.59) 33 } in present work a cohort of 64 employees was enrolled, and the training/testing sets was gotten by clarifying the theorem presented in equation 1 in every possible way, i.e., all the different possible sequences that combine the terms or clauses of the predicates’ extent ori-4, tri-4, rri-4, and cdri-5, a number given by the expression c p redicates−extent1 + ... + c p redicates−extentp redicates−extent , where c p redicates−extentp redicates−extent is a predicates-combination subset, with predicates-extent elements presented in equation 2. ∀(ex1, v a1, an1, esa1, qoi1, ..., ex4, v a4, an4, esa4, qoi4) (ori − 4(ex1, v a1, an1, esa1, qoi1), tri − 4(ex2, v a2, an2, esa2, qoi2), rri − 4(ex3, v a3, an3, esa3, qoi3), cdri − 5(ex4, v a4, an4, esa4, qoi4)) (1) {{ori − 4(ex1, v a1, an1, esa1, qoi1), tri − 4(ex2, v a2, an2, esa2, qoi2), rri − 4(ex3, v a3, an3, esa3, qoi3), cdri − 5(ex4, v a4, an4, esa4, qoi4)}, ...} ≈ ≈ {{ori − 4(0.31, 0, 0.69, 0.95, 0.69), tri − 4(0.53, 0, 0.47, 0.85, 0.47), rri − 4(0.34, 0, 0.66, 0.94, 0.66), cdri − 5(0.41, 0, 0.59, 0.91, 0.59)}, ...} (2) paradigmplus (2020) 1:3 36 ana fernandes et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 5: a creative view of the ann topology for esa assessment and a measure of its sustainability (qoi) for the best-case scenario with regard to the output of the ann, i.e., both the employee satisfaction assessment (esa) and a measure of its sustainability (qoi), for the best-case scenario they may be weighed in the mold by equation 3 leading to the ann depicted in figure 5. {{(esaori−4 + esatri−4 + esarri−4 + esacdri−4/4}, ...}bcs ≈ ≈ {{(0.95 + 0.85 + 0.94 + 0.91)/4 = 0.91}, ...}bcs {{(qoiori−4 + qoitri−4 + qoirri−4 + qoicdri−4/4}, ...}bcs ≈ ≈ {{(0.69 + 0.47 + 0.66 + 0.59)/4 = 0.60}, ...}bcs (3) it is therefore possible to describe an employee’s answers to the various items presented in the questionnaire in terms of the entropy they are associated with over a period of 5 (five) months (see figure 6). entropy follows a decreasing line from the first to the third month, which means that the employee increases his/her satisfaction with the safety procedures during this time. however, by the end of this month, it had inflected, i.e., the employee satisfaction with the preventive measures resulting from the pandemic crisis is declining. a view that culminates with a full break at the end of the 5th month. this process makes it possible to monitor and manage the evolution of the system; to be proactive through simulation, i.e., to foretell on the basis of observation, experience, or scientific reason; to act on the matters affecting its behavior, i.e., consider the answers to the items that contribute more to the deterioration of the system (i.e., have a higher entropic value or a higher value of exergy consumption). paradigmplus (2020) 1:3 assessing employee satisfaction in the context of covid-19 pandemic 37 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 6: variation in the entropy of the employees’ satisfaction with protection measures according to the perceptions of an employee for the best-case scenario 5.3.3 the worst-case scenario regarding the worst-case scenario, esa and qoi are evaluated in a similar way by the program presented in listing 5 listing 5: the make-up of the lp that establishes the esa implementation in the worst-case scenario 1 { 2 /* the s e n t e n c e b e l o w s t a t e s that the e x t e n t of p r e d i c a t e ori -4 is ⤦ ç b a s e d on the e x p l i c i t l y s p e c i f i e d c l a u s e s and t h o s e that ⤦ ç c a n n o t be d r o p p e d */ 3 ¬ ori − 4(ex, v a, an, esa, qoi) 4 ← not ori − 4(ex, v a, an, esa, qoi), 5 not exceptionori−4(ex, v a, an, esa, qoi) 6 7 /* the f o l l o w i n g s e n t e n c e s t a n d s for an a x i o m of ori -4 */ 8 ori − 4(0.31, 0.33, 0.36, 0.77, 0.36) 9 10 /* the s e n t e n c e b e l o w s t a t e s that the e x t e n t of p r e d i c a t e tri -4 is ⤦ ç b a s e d on the e x p l i c i t l y s p e c i f i e d c l a u s e s and t h o s e that ⤦ ç c a n n o t be d r o p p e d */ 11 ¬ tri − 4(ex, v a, an, esa, qoi) 12 ← not tri − 4(ex, v a, an, esa, qoi), 13 not exceptiontri−4(ex, v a, an, esa, qoi) 14 15 /* the f o l l o w i n g s e n t e n c e s t a n d s for an a x i o m of tri -4 */ 16 tri − 4(0.53, 0.36, 0.11, 0.96, 0.11) 17 18 /* the s e n t e n c e b e l o w s t a t e s that the e x t e n t of p r e d i c a t e rri -4 is ⤦ ç b a s e d on the e x p l i c i t l y s p e c i f i e d c l a u s e s and t h o s e that ⤦ ç c a n n o t be d r o p p e d */ 19 ¬ rri − 4(ex, v a, an, esa, qoi) 20 ← not rri − 4(ex, v a, an, esa, qoi), 21 not exceptionrri−4(ex, v a, an, esa, qoi) 22 paradigmplus (2020) 1:3 38 ana fernandes et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 /* the f o l l o w i n g s e n t e n c e s t a n d s for an a x i o m of rri -4*/ 24 rri − 4(0.34, 0.36, 0.30, 0.71, 0.30) 25 26 /* the s e n t e n c e b e l o w s t a t e s that the e x t e n t of p r e d i c a t e cdri -5 is ⤦ ç b a s e d on the e x p l i c i t l y s p e c i f i e d c l a u s e s and t h o s e that ⤦ ç c a n n o t be d r o p p e d */ 27 ¬ cdri − 5(ex, v a, an, esa, qoi) 28 ← not cdri − 5(ex, v a, an, esa, qoi), 29 not exceptioncdri−5(ex, v a, an, esa, qoi) 30 31 /* the f o l l o w i n g s e n t e n c e s t a n d s for an a x i o m of cdri -5*/ 32 rri − 4(0.41, 0, 0.59, 0.91, 0.59) 33 } in this case, regarding the output of the ann, i.e., both the esa and the qoi may be weighed as before as presented in equation 4 leading to the ann depicted in figure 7. {{(esaori−4 + esatri−4 + esarri−4 + esacdri−4/4}, ...}w cs ≈ ≈ {{(0.77 + 0.46 + 0.71 + 0.91)/4 = 0.71}, ...}w cs {{(qoiori−4 + qoitri−4 + qoirri−4 + qoicdri−4/4}, ...}w cs ≈ ≈ {{(0.36 + 0.11 + 0.30 + 0.59)/4 = 0.34}, ...}w cs (4) regarding the entropy associated over a period of 5 (five) months the same comments apply in this scenario, but the inflection and break points occur earlier (see figure 8). the software used to implement anns was the waikato environment for knowledge analysis figure 7: a creative view of the ann topology for esa assessment and a measure of its sustainability (qoi) for the worst-case scenario paradigmplus (2020) 1:3 assessing employee satisfaction in the context of covid-19 pandemic 39 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 8: variation in the entropy of the employees’ satisfaction with protection measures according to the perceptions of an employee for the worst-case scenario (weka), while maintaining the standard software parameters [23, 24]. to guarantee the significance of the results attained, 25 experiments were performed in all tests. in each simulation, the database was randomly split into two mutually exclusive partitions, i.e., the training set, with 2/3 of the data, used to build-up the model, and the test set, with the remaining cases to evaluate its performance. table 6 presents the confusion matrix for the ann model. the values shown in table 6 allow computing the model accuracy for training set (92.1%, i.e., 198 well classified in 215) and for test set (90.5%, i.e., 95 well classified in 105). in the classification process, high denotes esa higher than 0.85, medium stands for esa ranging in the interval 0.5...0.85, and low were assigned to esa lesser than 0.5. table 6: confusion matrix regarding ann model for employees’ satisfaction assessment target predictive training set test set low medium high low medium high low 41 1 0 16 0 0 medium 2 119 1 5 67 2 high 0 3 38 3 12 anns stand for computational tools that can be used in generalization processes, i.e., have a correct input-output mapping even if the inputs are different from the examples used to train the network. however, if the network over stamps the training data, it may capture some aspects that are present in the training data that are not applicable to the function being modeled. this phenomenon is known as overfitting and occurs when the network loses the ability to generalize [20]. however, considering that the accuracies of training and test sets are close, the selected network performed well in prediction of the output variables for an independent data set and, therefore, show no overfitting. 6 conclusions and future work employee satisfaction is an old but at the same time very topical issue, as it is closely related to productivity. the actual pandemic crisis brought new challenges for the corporate sector, which had to adapt to new requirements in order to avoid or at least minimize contagion. ensuring worker protecparadigmplus (2020) 1:3 40 ana fernandes et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . tion should be the responsibility not only of citizens, but also of organizations. however, it is difficult to judge as it involves dealing with employee satisfaction with different variables with complex relationships between them. in order to be able to assess the satisfaction of the employees in a water laboratory, a data acquisition and evaluation model was developed and experienced in practice. the focus was on an artificial intelligence/logic programming/entropy approach to data processing, system evolution and management (being the data collected through questionnaire’s answering), means that is brought into interrogation the immaterial part of this process, the emotions and people feelings. people is the most important asset in any organization. the proposed model offers a good accuracy of more than 90% and do not show overfitting. future work will consider expanding the study to a larger sample in order to assess the influence of other variables such as age, gender or academic qualifications of employees on their satisfaction with the preventive measures resulting from the actual pandemic crisis. in addition, the relationships between the conscientious and the unconscious and their use for assessing and predicting the development and management of employee satisfaction. authors’ information – ana fernandes is a phd student in biochemistry, master in environmental chemistry, and graduated in chemistry at the university of évora. she is senior technician in health and safety at work and is trainer, consultant and auditor of several entities, public and private, in the areas of certification of quality, environment, safety, health and safety at work and laboratory accreditation. she is also a professor at several higher education institutions and member of the technical commission 199-health information systems of the portuguese institute for quality. her current interests include quality assessment, health and safety at work, organizational efficiency and public health. – rui lima is since 2000 a professor and responsible for the laboratories of the escola superior da saúde at vale do ave, cespu, portugal. degree in food engineering, with a postgraduate degree in management of organizations – business management branch and master in entrepreneurship and innovation in the food industry by the polytechnic institute of viana do castelo, portugal. he is currently enrolled in the phd program in intelligent and adaptable software systems at the university of vigo, spain. the current areas of research and interest range from social entropy and its connection to chaos theory and artificial intelligence. – margarida figueiredo went to the university of évora, where she studied physics and chemistry and obtained her degrees in 1988. she joined the university of évora in 1990 and received her phd in chemistry in 2004. she is now auxiliary professor at the department of chemistry of university of évora. she is a researcher at the center for research in education and psychology and her currents interests include the study of methodologies for chemistry teaching, quality assessment in laboratories. – jorge ribeiro is adjunct professor in the school of technology and management polytechnic of the institute of viana do castelo (ipvc) – portugal, since 2006. he holds a phd (informatics engineering) in 2011 at the department of electronic and computation of the university of santiago de compostela, spain. he is affiliated with the adit-lab (applied digital transformation laboratory) of the polytechnic of the institute of viana do castelo. his main research interests are artificial intelligence, software engineering and knowledge discovery in databases and he has been author and co-author of some papers in the field of data mining, software engineering, knowledge representation, evolutionary systems and data mining using geographic information systems. – josé neves is full professor of computer science at minho university, portugal, and emeritus professor since 2019 at the same university. he also integrates the faculty of cespu (cooperative of higher education polytechnic and university, crl). he received his ph.d. in computer paradigmplus (2020) 1:3 assessing employee satisfaction in the context of covid-19 pandemic 41 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . science from heriot watt university, scotland, in 1983. his current research interests relate to the areas of logic programming, knowledge representation and reasoning, evolutionary intelligence and machine learning. lately it has been focusing on the area of computational sustainability, namely on the relationship between the disciplines of psychology, economics, computer science, mathematical logic and philosophy and their contributions to decision making in business environments. – henrique vicente went to the university of lisbon, where he studied chemistry and obtained his degrees in 1988. he joined the university of évora in 1989 and received his phd in chemistry in 2005. he is now auxiliary professor at the department of chemistry at the university of évora. he is a researcher at the requimte/laqv (university of évora) and at algoritmi research center (university of minho). his current interests include water quality control, lakes and reservoirs management, data mining, knowledge discovery from databases, knowledge representation and reasoning systems, evolutionary intelligence, and intelligent information systems. authors’ contributions – ana fernandes participated in the elaboration of the questionnaire, their validation, and implementation, as well as, in the computational features of the work. in addition, she contributed to the discussion of the results and to the writing of the article. – rui lima participated in the elaboration of the questionnaire, their validation, and implementation, as well as, contributed to the discussion of the results and to the writing of the article. – margarida figueiredo participated in the elaboration of the questionnaire, their validation, and implementation, as well as, contributed to the discussion of the results and to the writing of the article. – jorge ribeiro participated in the computational features of the work, as well as, contributed to the discussion of the results and to the writing of the article. – josé neves participated in the computational features of the work, as well as, contributed to the discussion of the results and to the writing of the article. – henrique vicente participated in the computational features of the work, as well as, contributed to the discussion of the results and to the writing of the article. competing interests the authors declare that they have no competing interests. funding this work has been supported by fct fundação para a ciência e tecnologia within the r&d units project scope: uidb/00319/2020 and uidb/50006/2020. references [1] portuguese government, “establishes exceptional and temporary measures regarding the epidemiological situation of the new coronavirus, covid–19, decree-law no. 10-a/2020 of 13 march, 2020,” 2020. paradigmplus (2020) 1:3 42 ana fernandes et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [2] g. correia, l. rodrigues, m. silva, and t. gonçalves, “airborne route and bad use of ventilation systems as non-negligible factors in sars-cov-2 transmission,” medical hypotheses, vol. 141, p. 109781, 2020. [3] world health organization, “laboratory biosafety guidance related to coronavirus disease 2019 (covid-19): interim guidance, 12 february 2020,” tech. rep., world health organization, 2020. [4] world health organization, “considerations in adjusting public health and social measures in the context of covid-19: interim guidance, 16 april 2020,” tech. rep., world health organization, 2020. [5] world health organization, “cleaning and disinfection of environmental surfaces in the context of covid-19: interim guidance, 15 may 2020,” tech. rep., world health organization, 2020. [6] k. alfes, a. d. shantz, c. truss, and e. c. soane, “the link between perceived human resource management practices, engagement and employee behaviour: a moderated mediation model,” the international journal of human resource management, vol. 24, no. 2, pp. 330–351, 2013. [7] e. m. berman, j. s. bowman, j. p. west, and m. r. van wart, human resource management in public service: paradoxes, processes, and problems. cq press, 2019. [8] y. georgellis, t. lange, a. i. petrescu, and r. simmons, “human resource management practices and workers’ job satisfaction,” international journal of manpower, vol. 29, pp. 651–667, 2008. [9] d. tengilimoğlu, a. zekioğlu, n. tosun, o. işık, and o. tengilimoğlu, “impacts of covid-19 pandemic period on depression, anxiety and stress levels of the healthcare employees in turkey,” legal medicine, vol. 48, p. 101811, 2020. [10] s. x. zhang, s. sun, a. a. jahanshahi, a. alvarez-risco, v. g. ibarra, j. li, and r. m. patty-tito, “developing and testing a measure of covid-19 organizational support of healthcare workers– results from peru, ecuador, and bolivia,” psychiatry research, vol. 291, p. 113174, 2020. [11] d. d. bajrami, a. terzić, m. d. petrović, m. radovanović, t. n. tretiakova, and a. hadoud, “will we have the same employees in hospitality after all? the impact of covid-19 on employees’ work attitudes and turnover intentions,” international journal of hospitality management, p. 102754, 2020. [12] d. p. stergiou and a. farmaki, “ability and willingness to work during covid-19 pandemic: perspectives of front-line hotel employees,” international journal of hospitality management, vol. 93, p. 102770, 2020. [13] h. s. jung, y. s. jung, and h. h. yoon, “covid-19: the effects of job insecurity on the job engagement and turnover intent of deluxe hotel employees and the moderating role of generational characteristics,” international journal of hospitality management, vol. 92, p. 102703, 2021. [14] h. shin and j. kang, “reducing perceived health risk to attract hotel customers in the covid-19 pandemic era: focused on technology innovation for social distancing and cleanliness,” international journal of hospitality management, vol. 91, p. 102664, 2020. [15] j. c. ferreira maia neves, “a logic interpreter to handle time and negation in logic data bases,” in proceedings of the 1984 annual conference of the acm on the fifth generation challenge, pp. 50–54, 1984. [16] a. c. kakas, r. a. kowalski, and f. toni, “the role of abduction in logic programming,” handbook of logic in artificial intelligence and logic programming, vol. 5, pp. 235–324, 1998. paradigmplus (2020) 1:3 assessing employee satisfaction in the context of covid-19 pandemic 43 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [17] j. neves, n. maia, g. marreiros, m. neves, a. fernandes, j. ribeiro, i. araújo, n. araújo, l. ávidos, f. ferraz, et al., “entropy and organizational performance,” in international conference on hybrid artificial intelligence systems, pp. 206–217, springer, 2019. [18] t. wenterodt and h. herwig, “the entropic potential concept: a new way to look at energy transfer operations,” entropy, vol. 16, no. 4, pp. 2071–2084, 2014. [19] a. fernandes, m. figueiredo, a. dias, j. ribeiro, j. neves, and h. vicente, “a case-based approach to assess employees’ satisfaction with work guidelines in times of the pandemic,” in international conference on applied informatics, pp. 183–196, springer, 2020. [20] s. s. haykin et al., “neural networks and learning machines.,” 2009. [21] p. cortez, m. rocha, and j. neves, “evolving time series forecasting arma models,” journal of heuristics, vol. 10, no. 4, pp. 415–429, 2004. [22] y. shang, “subgraph robustness of complex networks under attacks,” ieee transactions on systems, man, and cybernetics: systems, vol. 49, no. 4, pp. 821–832, 2017. [23] e. frank, m. hall, and i. witten, “the weka workbench. online appendix for data mining: practical machine learning tools and techniques,” 2016. [24] m. hall, e. frank, g. holmes, b. pfahringer, p. reutemann, and i. h. witten, “the weka data mining software: an update,” acm sigkdd explorations newsletter, vol. 11, no. 1, pp. 10–18, 2009. paradigmplus (2020) 1:3 introduction related work health risks hospitality industry fundamentals logic programming a thermodynamics approach to knowledge representation and reasoning artificial neural networks material and methods employees questionnaire methodology results and discussion sample characterization a thermodynamic approach to data attainment and processing the esa implementation – a logic programming line the formal framework the best-case scenario the worst-case scenario conclusions and future work vol. 1, no. 1, pp 1 21, 2020 doi: 10.55969/paradigmplus.v1n1a1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . evaluation of the bias in the management of patient’s appointments in a pediatric office cinthia vegega1,�, pablo pytel1, and maría florencia pollo-cattaneo id 1 1information system methodologies research group, universidad tecnológica nacional facultad regional buenos aires. argentina cinthiavg@yahoo.com.ar, {ppytel,flo.pollo}@gmail.com abstract the application of machine learning algorithms must always take into account the objectives set within the project, the characteristics of the domain where the project will be carried out and the data available to use. given this, it is essential before collecting data considered as representative of the problem to be solved, because otherwise there may be hidden biases in the data and these may solve a different problem from the one intended. in this context, the aim of this work is to apply a process based on the gridding method that allows the analysis of the features of the data to be used. this process is applied to the historical data of a pediatric medical office where it is sought to implement an intelligent system that allows to predict the number of normal and overshift appointments for a particular date and time, since it is desired to hire, when necessary, another pediatric doctor to assist in the care of patients. keywords: intelligent systems ⋅ machine learning ⋅ training data ⋅ repertory grid ⋅ bias received: 23 february 2020 ⋅ accepted: 1 april 2020 ⋅ published: 9 april 2020. 1 introduction machine learning is considered to be the discipline that studies and models learning processes with their multiple manifestations in order to be able to be transferred to computers [1, 2]. the aim is to reproduce aspects of the behavior of intelligent beings to facilitate the implementation of software systems in complex environments [3] by building models from the observed data [4]. the software systems thus generated are called intelligent systems [5, 6] and can be applied in different domains to solve different types of problems. since the beginning of this century, its use has spread rapidly and it is often a more attractive alternative than building models manually, although "automatic learning is not magic, you cannot get something out of nothing" [7], since in order to apply any of the algorithms it is essential to have data. one of the types of problems where machine learning has been applied has to do with the construction of models capable of inferring from historical data the dependencies between the past value and the short-term future, which are the so-called predictive models. as indicated [8], predicting the future is one of the most important and difficult tasks in the applied sciences. one of the main reasons is that predictive models start from the behavior of historical data, so they must assume that the future will be the same (or similar) to the past [9]. in the event that some behavior changes, the generated model will become inefficient. therefore, it is this article is distributed under the terms of the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. https://doi.org/10.55969/paradigmplus.v1n1a1 https://orcid.org/0000-0003-4197-3880 2 cinthia vegega et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . essential to have a large amount of data representative of the domain of the problem that allows the model to generate better predictions. likewise, when the data are not representative enough, they are said to have a bias. this term is used to speak of something biased or twisted [10], so it is used in psychology to refer to the tendency or prejudice of a person to perceive and interpret reality, thus generating a distortion of it. in statistics, bias is used to refer to the difference between the value generated by the model and the expected value [11]. in the context of predictive models something similar happens. if the data presents a bias, then there is a risk of generating an intelligent system that is not based on reality and produces erroneous results [7]. in other words, you could be training the algorithms to solve a different problem from the one you want to solve. for this reason, it is essential to know in advance the biases associated with the data and the intelligent system so that they can be understood by future users to avoid misunderstandings and situations of discrimination [12]. it is not rare that developers with the best intentions can involuntarily produce intelligent systems with prejudicial results, because even they may not understand enough of the problem, its context and the data to prevent unintended results. the worst thing about this scenario is that the bias may be so subtle that it goes undetected during testing. if this system is then put into operation and users come to rely blindly on the long-term results, it could lead to situations of sexism, racism and other forms of discrimination [13]. in this context, an adaptation of the repertory grid has been proposed in [14] to evaluate the collected data with the assistance of the available domain experts and determine whether it can be used. likewise, in [15] the proposed method has been applied to a higher education institution course data to predict the students’ performance throughout the course and identify their strengths and weaknesses. in this work, the method is applied to another case study for the management of patient’s appointments in a pediatric office in order to evaluate whether they are sufficiently representative to implement a future intelligent system. for this purpose, section 2 presents the problem identification, section 3 describes the repertory grid technique and section 4 describes the data gathered and its context, whose evaluation is dealt with in section 5. section 6, implements a prototype of the intelligent system based on the data evaluated. finally, section 7 presents the conclusions and the future line of work. 2 problem identification obtaining the data for the construction of predictive models is not a trivial task. for example, take the case of determining the amount of historical information needed to obtain the best results [16]. according to [17], the usual answer to the question "how much data are needed" is "as much as possible". the more data you have, the better you will be able to identify the structure of the model and the patterns used for prediction, but in practice it is essential to put some limits on it. although there are publications [18, 19] that indicate minimum requirements for the amount of data to be applied, these are considered over-simplified because they ignore aspects such as the underlying random variability of the data. therefore, in order to define the amount of data to be used, it is first necessary to identify the available data sources and understand their characteristics. only then, it will be possible to collect data that fully represent the problem to be solved [20]. the same happens when determining the appropriate attributes to be selected as input and output variables of the model [16]. to this end, in addition to analyzing the sources of historical information, the characteristics of the domain, the objectives set and, often, the technology used for the construction of the predictive model must also be considered. it may also happen that the historical data available in computerized repositories are not enough, so other sources should be used, such as the opinion of experts in the field [17]. likewise, when the data is not sufficiently representative and presents a bias, there is a risk of generating an intelligent system that is not based on reality and produces erroneous results [7]. in other words, you could be training the algorithms to solve a different problem from the one you want to solve. however, it is not true that this is always a bad thing. according to tom mitchell’s principle paradigmplus (2020) 1:1 evaluation of the bias in the management of patient’s appointments in a pediatric office 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . of "the futility of bias-free learning" [21], biases are necessary for algorithms to work. by removing them it may seem that a desirable goal is achieved, but in truth the result becomes virtually useless since an ’unbiased’ intelligent system loses the ability to generalize and process new examples. so, although in ordinary life prejudice is a pejorative word, since preconceived notions are bad, in this case, preconceived notions are absolutely necessary for the algorithm to learn [22]. in any case, as mentioned in the previous section, it is essential to know in advance the biases associated with the data and the intelligent system so that they can be understood by future users to avoid misunderstandings and discrimination situations [12]. an illustrative example of this situation is beauty ai [23], which has been publicized as the "first beauty contest judged by robots" and to which approximately 6,000 women from more than 100 countries around the world have registered. according to its creators, this intelligent application seeks to eliminate the prejudices of human jurors by using objective factors of a contestant’s image (such as the amount of wrinkles and facial symmetry) to identify the most attractive ones. but, according to [24], at the end of the competition the result was not as expected: of the 44 winners, practically all were white (very few were of asian origin, and only one had dark skin), although the developers claim that they did not consider skin color as a sign of beauty. this is because during the training the algorithm had a large number of photos of caucasian women but very few of women of color, that is, there was a significant bias in the data. to solve this problem, the application had to be retrained, including a greater variety of photos and generating a new version. another case, more serious, is the software applied in the ’predict policing’ initiative [25], which seeks to prevent crime by determining where the police should patrol according to the prediction of the occurrence of future crimes. according to its critics [26], many of the predictions generated by the algorithms tend to fail because the data provided by the police is incomplete, partial and/or erroneous [27]. in addition, the number of failures increase as the system is used. consider the case in which police officers detect and stop a minor crime in the area they were sent to patrol, which according to [28] will always happen because the police tend to think that there will be a crime and therefore suspect of everyone in the area. the crime data recorded will then be used by the algorithm to increase the risk of the area. in the long run, this could result in more patrol assignments to that area leaving others that may actually be at similar or higher risk with few or no patrols. something similar has been detected about software used to assess the risk of recidivism in offenders [29]. in [30], it was found that a black defendant was twice as likely to commit future crimes and was therefore assigned a high risk, while white defendants were considered to be low risk. since this information is used by the judge to determine whether they were granted parole, it has generated a significant disproportion in the prison population. one of the main issues in the implementation of intelligent systems has to do with the steps associated with collecting the data, integrating, cleaning and pre-processing it. it is therefore essential to identify and understand them first [31]. in [12], the need for intelligent systems to avoid undesired behavior and generate evidence that involuntary failures are unlikely is emphasized. to this end, it is essential to have "transparency" not only about the data and algorithms involved, but also in explaining how the results have been generated. therefore, progress must be "continued" by making ai (in general) and machine learning (in particular) a mature field of engineering so that they can create "predictable, reliable, robust and safe systems". consequently, this work aims to assist the developers of an intelligent system, at the beginning of the project, in identifying sources of bias in the data and thus reducing their impact. in the initial tasks of these projects, work is done on the steps associated with collecting the data, integrating it, cleaning it up and pre-processing it. to do this, it is essential to identify and understand the data first [31], which is not a trivial task. the sources of historical information must be analyzed, and the characteristics of the domain, the objectives set and the expectations of users must also be considered. in addition, historical data available in computerized repositories may be insufficient, so other sources, such as expert opinion, should also be used [17, 22]. however, experts are usually unable or unwilling to provide their knowledge, and more effective indirect techniques are needed to educate paradigmplus (2020) 1:1 4 cinthia vegega et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . and represent that knowledge [6]. for this reason, this proposal considers the possibility of applying a semi-automatic procedure to generate a representation of the available data and allow experts to analyze and interpret it more easily. 3 repertory grid the repertory grid technique has been defined by psychologist george kelly in [32] and is based on the theory of personal constructors. a construct is, according to the rae [33], a "bipolar descriptive category" that people use to organize "data and experiences from their world", which kelly considers as "a way in which two or more things are similar and therefore different from a third or fourth thing" [34]. so, "personal constructs" are characteristics of objects or elements that allow a person to make judgments [35] and interpret the world around him. in this sense, the grid applies these bipolar characteristics to generate an objective representation of the mental image by which a person distinguishes between similar and different elements [36]. to this end, it seeks to "go beyond words" [34] by processing the subjective data provided by the person through simple operations [37]. the basic steps of a grid are divided into the following five stages [34, 36, 38]: 1○ identification of the elements: seeks to identify a homogeneous and representative set of conceptual elements within each category involved in knowledge. 2○ identification of characteristics: determines the list of bipolar characteristics that can be attributed to the elements identified above. 3○ design of the grid: with the elements and characteristics identified, a two-dimensional matrix (the "grid") is generated, where the elements are located in the columns and the characteristics in the rows. for each element/characteristic intersection, the person must enter a numerical value, and there are three ways to construct a grid: – dichotomous grid: the values that are assigned are binary, depending on whether the element has (1) or not (0) each characteristic. – classification grid: it is assigned the logical position or ranking that an element has on each characteristic. – evaluation grid: the values are defined as the degree of satisfaction with which the element in question covers each characteristic. 4○ formalization: with the values assigned in the grid, it is proceeded to analyze the relationships between the elements and the characteristics independently of each other: – classification of the elements: it is a matrix of distances between the elements is generated using the manhattan distance formula [39] with the values of each pair of columns. with these distances, the elements start to be grouped, always taking the least distant ones (that is, with the smallest value). when all the elements are grouped, each of the groups will be represented with an ordered tree. – classification of the characteristics: since the characteristics are bipolar, two types of distance must be calculated: distance 1 using the manhattan distance between the rows of the grid and, distance 2, using the manhattan distance between one row of the grid and another row of its opposite (in which the complements of the grid values are assigned). then, these distances are unified by taking the smallest value between each combination of characteristics. finally, the same steps as for the element classification are applied, generating the corresponding ordered tree. paradigmplus (2020) 1:1 evaluation of the bias in the management of patient’s appointments in a pediatric office 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5○ analysis of the results: with the ordered trees already generated, it is proceeded to carry out their corresponding analysis and interpretation to determine the number of groups and the similarities between them. finally, these results are then presented and discussed together with the person who provided the information for the grid. the application of these steps can be seen in [38]. although repertory grid was originally defined as a means to help mentally ill people become aware of inconsistencies in their own scales of values [34, 38], it has subsequently been successfully applied in other domains [40], such as computer science, marketing, business administration, engineering and tourism, among others [41]. in this sense, the acquisition of knowledge for the construction of expert systems is significant [32, 42]. since experts in the field feel more comfortable (and more precise) when they are able to use their own terminology, repertory grid allows for the acquisition of knowledge in a more natural and flexible way than other techniques [35]. in addition, since a grid can analyze any topic, knowledge can be acquired on a wide range of subjects. another application can be found in the "theorise-inquire" technique [43], where the grid is used to evaluate experts’ "hunches" about the quality of available data sources. thus, it is possible to test whether the information registered in the database is in accordance with the expectations of the domain experts (i.e. complete and without errors). while these objectives are similar to those of the procedure proposed above, when looking at an example of this technique in [44], on the analysis of the databases of a large retail chain in the united kingdom, it can be seen that it works differently. in "theorise-inquire", repertory grid is used in its traditional sense, i.e. to "make explicit the knowledge of the experts and then use that knowledge to direct the data analysis" [43]. in other words, it is the experts who provide the values of the grid corresponding to the situations they have experienced (elements) and their distinctions from each other (characteristics). the results of this grid are then associated with the available data so that the experts’ theories can be confirmed. in contrast, in the procedure proposed above, the grids are generated directly from the data collected and then the results generated by the formalization of the grid are contrasted with the experts’ view. 4 context case study and collect data description the case study is conducted within a pediatric medical office, which sees children from birth to their teens on their 18th birthday. office appointments are provided online, with the exception of prenatal or first-time telephone consultations. despite not having an on-call service, unscheduled consultations from their own patients are often attended to. appointments for such consultations (called "over-shifts") are granted by telephone on the same day as they are required. such over-shifts appointments are discharged by the secretary the same day they are requested, since if an over-shift is required, the office must be called to check availability. the over-shifts appointments are registered between the normal appointments. for example, between the 9:00 a.m. and 9:30 a.m. normal appointments there may be an over-shift appointment at 9:15 a.m. in this context, the aim is to implement an intelligent system that allows for the prediction of the number of normal and over-shift appointments for a particular date and time, given that it is desired to hire, when necessary, another pediatrician to assist in the care of the patients. for this reason, it is considered essential that the system provides results that are consistent with the management of office appointments. in addition to collecting the historical data that will be applied to build the intelligent system, it is necessary to identify the general characteristics of the domain where the prediction is made. moreover, it is essential to detect situations or events to be considered for which there is no data or the data available is not representative. otherwise, the system could generate incorrect predictions, leading the physician to make wrong decisions. as for the data collected, the appointments are granted from november 2017 to date were obtained. previously, the management of the appointments was always in charge of the secretary of the paradigmplus (2020) 1:1 6 cinthia vegega et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . office, who from a phone call of the patient (in this case, mother, father or guardian, being a minor), recorded the appointment in a paper agenda. for this reason, as of november 2017, a computer system is being used where the patient can obtain his/her appointment directly through a web page with the additional option of being able to cancel it. the data was extracted directly from the computer system database in a comma-delimited text file (csv) containing the following ten attributes: – date: full appointment date, the format of which is dd/mm/yyyy (day/month/year). – hour: appointment time, with format hh:mm (hour:minutes). – id_patient: patient’s identifier (numerical data type). – surname: patient’s last name. – name: patient’s name. – id-specialist: identifier of the pediatrician treating the patient (type of numerical data) – specialist: name of attending pediatrician. – box: number of the office where the patient is treated. – observations: observations that the patient can add when booking the appointment. – status: appointment status (type of numerical data) the possible states are the following: ○ 1: "available" ○ 2: "booked" ○ 3: "waiting" ○ 4: "attending" ○ 5: "finished" ○ 6: "suspended" as for the status of an appointment, its management starts with the creation of the appointment, which is in charge of the office secretary. when an appointment is created, it is created with "available" status. patients log in to the office website and book their appointments at their convenience. each appointment that is booked by a patient is set to "booked" status. once the patient arrives at the office on the day his or her appointment is booked, the secretary records that the patient has arrived, so the appointment goes to "waiting" status. when the pediatrician attends to the patient, the appointment changes to "attending" status and at the end it changes to "finished" status. at the end of each day, if there are appointments that became available because patients did not show up or were not booked, the secretary records this situation and those appointments are passed to "suspended". the patient may cancel an appointment at any time. this is done through the appointment notification email that is sent to the patient’s personal email address. when patients cancel their appointments, the appointments return to "available" status. 5 results of the application of the repertory grid method from the available data obtained for the case study, it is analyzed if these data are representative of the business to build the intelligent system, that is, if the data are sufficient and appropriate to consider all the possible situations of the problem to be solved. this is done by applying the method proposed in [14]. this process is based on the repertory grid technique described previously in section 3, but paradigmplus (2020) 1:1 evaluation of the bias in the management of patient’s appointments in a pediatric office 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . presents some differences with respect to the original technique. the main difference is that the grids are generated directly from the data collected and then the results generated by their formalization are contrasted with the vision of the experts or stakeholders in the business. in addition, three grids are used: one corresponding to the elements (which allows the evaluation of already known classes) and, two for the characteristics (one direct and one opposite, to evaluate the rest of the attributes). therefore, it is necessary to first carry out a set of tasks related to the collection and preparation of data to generate the grids, which will then be processed to generate the trees to be interpreted. the results to evaluate the data are presented below. – phase a: design of the grids ○ activity a.1data preparation data preparation tasks including formatting, cleaning and integration of data are carried out. in this context, the following decisions are taken according to each attribute available in the csv file: ∎ box: is not taken into account as it always takes the same value. ∎ observations: is not taken into account because it is empty in all records. ∎ id_patient: cannot be used for confidentiality. ∎ surname: cannot be used for confidentiality. ∎ name: cannot be used for confidentiality. ∎ specialist: is not taken into account as it is considered to be redundant with the specialist’s identifier, so only the id-specialist attribute is used. ∎ status: since appointment records have been found that have not been assigned a final status ("finished" or "suspended"), it is adjusted as follows: · for the 94 records with "booked" status, the patient is considered not to have attended the appointment, so they are moved to the suspended status. · in contrast, the 40 records with "waiting" status and the 94 records with "attending" status are considered to have been attended to by the specialist and are therefore changed to the finished status. both decisions were consulted with the physician responsible for the office. ∎ date: this attribute is split in two: date-month (containing the month number) and date-day (that contains the day number). the year of the date is not considered, since the prediction will be made in the future. a new attribute appointmenttype is created to indicate whether or not it is an over-shift, assigning the values of "appointment" and "overshift-appointment". this attribute is calculated considering if there is another appointment for the same specialist with the same day, month, year, hour and a difference of minutes less than 15 (decision consulted with the office secretary). ∎ a new schedule attribute is created to indicate whether it is morning, midday or evening appointment. this generates a file containing the following seven attributes: date-month, date-day, id-specialist, schedule, status, appointment-type, of which the first four are numerical and considered as input attributes and the last two are combined to form the target attribute. as a result of this conjunction. the values: "appointment-finished", "overshift-appointment-finished", "appointment-suspended" and finally "overshift-appointment-suspended" are obtained. ○ activity a.2data segmentation in this task, the numerical attributes are segmented. the data is imported into the tanagra tool [45] and the kohonen som neural network [19] are used. with a distribution of 3x3 paradigmplus (2020) 1:1 8 cinthia vegega et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . neurons (i.e. with a maximum requested of 9 clusters), a distribution has been reached that has a cluster with less than 10 tuples ("c_som_2_3"), as can be seen in the sheet "res kohonen" (result of kohonen)1. also, a new attribute called cluster_som_1 has been added to the data indicating the cluster identifier assigned to each tuple. this new structure, which is included in the sheet "datos segmentados" (segmented data), will be used in subsequent tasks. table 1 shows the number of tuples per cluster. table 1: number of tuples assigned per cluster cluster cluster id number of tuples nº 1 c_som_1_1 1397 nº 2 c_som_1_2 1125 nº 3 c_som_1_3 1354 nº 4 c_som_2_1 1891 nº 5 c_som_2_2 1053 nº 6 c_som_2_3 0 nº 7 c_som_3_1 995 nº 8 c_som_3_2 1500 nº 9 c_som_3_3 21 ○ activity a.3design of the elements grid using the segmented data obtained in the previous task, the element grid is generated. it is a 4x9 matrix since it has four columns corresponding to the values of the class with values "appointment-finished", "overshift-appointment-finished", "overshift -appointment-suspended", and "appointment-suspended", as well as, nine rows corresponding to the clusters "c_som_1_1", "c_som_1_2", "c_som_1_3", "c_som_2_1", "c_som_2_2", "c_som_2_3", "c_som_3_1", "c_som_3_2", and "c_som_3_3"). to complete the matrix values, the steps proposed in [14] are applied. to illustrate these steps, the definition of the value v12 corresponding to class "overshift-appointment-finished" and cluster "c_som_1_1" is presented below as an example: 1○ the number of tuples is determined in the segmented data for the cluster "c_som_1_1" and class "overshift-appointment-finished", which is equal to 706. 2○ the total number of tuples is defined for "overshift-appointment-finished" class that is equal to 4002. 3○ the membership percentage is calculated using equation 1. p12 = t12 ∑9i=1 (ti2) = 706 4002 = 0.1764 (1) 4○ the percentage obtained using equation 2 is formatted. v12 = rounding up [p12 ⋅ 10] = rounding up [1.764] = 2 (2) 5○ then, 2 is recorded as the v12 value in the matrix corresponding to the class and the cluster. in the same way, the rest of the values are completed, as can be seen in the sheet "parrilla de elementos" (elements grid), obtaining as a result the grid shown in table 2. 1https://github.com/inteligenciaartificialutn/paradigm paradigmplus (2020) 1:1 https://github.com/inteligenciaartificialutn/paradigm evaluation of the bias in the management of patient’s appointments in a pediatric office 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 2: elements grid appointment overshiftovershiftappointment -finished appointment appointment -suspended -finished -suspended c_som_1_1 1 2 1 1 c_som_1_2 1 1 2 1 c_som_1_3 2 1 1 2 c_som_2_1 2 2 2 2 c_som_2_2 1 1 2 2 c_som_2_3 0 0 0 0 c_som_3_1 1 1 0 1 c_som_3_2 2 2 1 1 c_som_3_3 0 0 0 0 ○ activity a.4attribute weighting in this task, the segmented data of activity a.2 are again used, using the tanagra tool [45] to perform the weighting of the interdependence between the generated clusters and the data attributes. this is done by generating conditional probabilities through the näive bayes algorithm, as can be seen in the sheet "res naivebayes" (result of näive bayes). ○ activity a.5design of the characteristics grids from the conditional probability tables obtained from the previous task, the two characteristics grids (direct and opposite) are generated. these have a 9x4 structure since there are nine cluster values indicated as columns, and 4 attributes as rows. as you can see in the sheet "parrilla de caracteristicas" (characteristics grid), we proceed in this way with all the combinations obtaining the grids indicated in tables 3 and 4. table 3: direct characteristics grid c_ so m _1 _1 c_ so m _1 _2 c_ so m _1 _3 c_ so m _2 _1 c_ so m _2 _2 c_ so m _2 _3 c_ so m _3 _1 c_ so m _3 _2 c_ so m _3 _3 date-month 0 1 10 10 0 0 0 10 10 date-day 9 0 6 10 10 0 0 0 3 d-specialist 10 10 10 10 10 0 10 10 0 schedule 0 4 7 0 4 0 0 0 0 – phase b: formalization and analysis of the grids ○ activity b.1classification of the elements from the element grid obtained in activity a.3, the distances between the columns (i.e., the elements or classes) are calculated using the manhattan distance [39] over the values in table 2. in this way, the element distance matrix shown in table 5 is obtained. since the absolute differences between the classes are considered, only the distances above the diagonal are indicated because the distances below it are equal. paradigmplus (2020) 1:1 10 cinthia vegega et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 4: opposite characteristics grid c_ so m _1 _1 c_ so m _1 _2 c_ so m _1 _3 c_ so m _2 _1 c_ so m _2 _2 c_ so m _2 _3 c_ so m _3 _1 c_ so m _3 _2 c_ so m _3 _3 date-month 10 9 0 0 10 0 10 0 0 date-day 1 10 4 0 0 0 10 10 7 d-specialist 0 0 0 0 0 0 0 0 10 schedule 10 6 3 10 6 0 10 10 10 table 5: matrix of distances between elements appointment overshiftovershiftappointment -finished appointment appointment -suspended -finished -suspended appointment 2 5 2 -finished overshift5 4 appointment -finished overshift3 appointment -suspended appointment -suspended from this distance matrix, some groupings are made with the minimum distance criterion. in this case, the minimum distance is 2; then, the classes "appointment-suspended", "appointment-finished", and "overshift-appointment-finished" have been joined. later, the distance matrix is updated, as shown in table 6. in this way, only the "overshift-appointment-suspended" and the "[appointment-finished, table 6: matrix of distances between elements after the first cluster overshift[appointment-finished, appointment appointment-suspended, -suspended overshift-appointmentfinished] overshift3 appointment -suspended [appointment-finished, appointment-suspended, overshift-appointmentfinished] paradigmplus (2020) 1:1 evaluation of the bias in the management of patient’s appointments in a pediatric office 11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . appointment-suspended, overshift-appointment-finished]" group needs to be joined to finish (the graphic representation of this is shown in activity b.3). ○ activity b.2classification of the characteristics in a similar way to the previous task, the distance matrix is generated for the attributes or characteristics of the grids obtained in task a.5. however, since there are two grids, a direct one associated to the first pole of the attributes and an opposite one associated to the second pole, in this case for each combination it will be necessary to calculate two distances (d1 and d2). in this way, the distance matrix d1 and d2 of the characteristics is obtained, which can be seen on the sheet "distancias de caracteristicas" (characteristics distances). as it can be seen, the values of d1 are located above the diagonal and those of d2 below it. since a single distance is necessary to make the groupings, d1 and d2 must be unified, taking the lowest value for each combination. this means that, for each combination of attributes, the minimum is taken between the value below and above the diagonal. then, the values below the diagonal are discarded, thus obtaining the unified distance matrix (included in the same sheet of the spreadsheet). finally with the unified matrix, the corresponding groupings are carried out with the minimum distance criterion as it was done with the elements in the previous task (the graphic representation of this is shown in activity b.3). ○ activity b.3interpretation of the results from the groupings obtained in the tasks b.1 and b.2, both the ordered tree of elements shown in figure 1 and the ordered tree of characteristics in figure 2 are generated, which are also available in the sheet "interpretación elementos" (interpretation of elements) and the sheet ’interpretación características’ (interpretation of characteristics). the interpretation of these trees is included below: ∎ analysis of the ordered tree of elements: two groups have been generated in this tree. thus, both suspended normal appointments ("appointment-suspended") and finished normal appointments ("appointment-finished"), as well as the finished over-shift appointments ("overshift-appointment-finished") have a more similar behavior than the suspended over-shift appointments ("overshiftappointment-suspended"). ∎ analysis of the ordered tree of characteristics: in this tree 3 groups have been generated. as a first grouping, the specialist physician (id-specialist) is related to the schedule in which he attends (schedule). then, this grouping is related to the month (date-month) and finally, to the day of the appointment (date-day). figure 1: ordered tree of elements paradigmplus (2020) 1:1 12 cinthia vegega et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 2: ordered tree of characteristics ○ activity b.4discussion of the results as a last task, a meeting is held with the pediatrician and the office secretary who take the role of experts in the domain to discuss the results generated in the previous tasks that are interpreted in activity b.3. in the first instance, the analysis of the ordered tree of elements is presented to the pediatrician and the office secretary, explaining the meaning of the tree together with its interpretation. in view of the relationship between normal appointments with status completed paradigmplus (2020) 1:1 evaluation of the bias in the management of patient’s appointments in a pediatric office 13 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . and suspended together with completed over-shift appointments, they mention that this relationship is correct, given that the behavior of over-shift appointments is similar to the behavior of normal appointments. the only difference is that the over-shift appointments are requested on the same day by the patient and not in advance as the shifts. on the other hand, they agree that it is correct that suspended over-shift appointments are not within the same relationship because it is very rare for over-shift appointments to be canceled by patients. in general, unless the patient requires a very urgent consultation and must attend an appointment, patients who ask for over-shift appointments attend the office at the time indicated by the secretary. then, the analysis of the attribute groupings is continued. for this purpose, the ordered tree of characteristics is presented, together with its interpretation. in view of this, they express that they also agree on the relationships generated, since the specialist who attends in the office always has a certain schedule. in this case, within the office, there are only two specialists but the objective is to incorporate even more in determined schedules. likewise, there are times when the specialists take turns (for example, during vacation periods) so it is considered correct to relate this to the month and finally to the day of the month in which they attend. likewise, the doctor tells us that he would have liked to know the season of the year, because depending on the season there may be more or less appointments. as the season can be calculated with the existing data, it is not necessary to carry out a new analysis and the interpretation of the tree is considered correct. it is therefore concluded that the version of the data is representative of the behavior of the office’s normal and over-shift appointments. from these conclusions, the version of the data used is representative of the behavior of the appointments of the office, therefore, it is the one that can be used for the implementation of the intelligent system. 6 implementation of the intelligent system prototype from the data considered as representative and the problem to be solved within the case study, an initial prototype of the intelligent system is built. in this case it is very important to carry out a prediction of the number of normal and over-shift appointments with precision because it is critical to ensure the success of the model, but it is not necessary to know how the results generated were obtained. in other words, it is not necessary for the network to explain or justify the results obtained. therefore, in this case, it has been determined that the most suitable architecture for the intelligent system corresponds to multi-perception artificial neural networks (ann) with training by backward error correction [16], better known as backpropagation ann [46]. this type of network applies supervised training and only has forward connections between its neurons which are organized in several layers [47]: – the first layer is the input layer for receiving the values from the outside. – the last layer is the output layer for returning the results generated by the network, and – he neurons located in the intermediate layers are called hidden or processing layers (in a network there may be no hidden layer, a hidden layer or more than one). before starting work on ann modeling, it is necessary to transform the available data to a format that best suits this architecture. then, the values are converted into numerical values so that they can be used. since it is desired to be able to estimate the number of normal and over-shift appointments for a given date and time, taking into account the appointments actually attended (i.e. appointment with a "finished" status) such as those that were requested but then canceled (in a "suspended" status), the following input and output attributes are defined for ann: paradigmplus (2020) 1:1 14 cinthia vegega et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . – input attributes: ○ date-month, corresponding to the month of the date to be estimated. ○ date-day, which corresponds to the day of the month of the date to be estimated. ○ date-ssn, which indicates the season of the year corresponding to the date to be estimated (where 1 corresponds to "summer", 2 to "fall", 3 to "winter", and 4 to "spring"). ○ date-schdl, which indicates the time of the date to be estimated (where 1 corresponds to "morning", 2 to "noon", and 3 to "evening"). – output attributes: ○ n-app-finish, which indicates the number of normal appointments with finished status for that date and time. ○ n-osa-finish, which indicates the number of over-shift appointments with finished status for that date and time. ○ n-app-susp, which includes the number of normal appointments with suspended status for that date and time. ○ n-osa-susp, which indicates the number of over-shift appointments with suspended status for that date and time. an extract of the prepared data can be seen in table 7. table 7: extract from the prepared data d a t em o n t h d a t ed a y d a t ess n d a t esc h d l n -a pp -f in is h n -o sa -f in is h n -a pp -s u sp n -o sa -s u sp 4 11 2 3 0 1 2 4 5 17 3 1 11 2 7 8 7 16 3 3 4 1 0 1 9 31 4 2 1 1 1 0 11 1 1 1 9 5 0 4 once the data is prepared, the initial model is built. for this purpose, the neat4j framework2 is used, which implements in java the "neuroevolution of augmenting topologies" or neat [48] algorithm for the construction of ann using evolutionary algorithms. in this case, a fitness function provided by the framework called "mse neat fitness function" is used to minimize the error when calculating the square average of the difference between the desired output and the one generated by the network. due to the requirements of this function, the values of the attributes have had to be reformatted into decimal numbers (when divided by 100), as can be seen in table 8. this data is then stored in a new csv file to be accessed from the framework. finally, the data from the new csv file are supplied to neat4j, so that the latter may begin to evolve possible ann topologies to generate the quantities corresponding to the date to be estimated. 2http://neat4j.sourceforge.net/ paradigmplus (2020) 1:1 http://neat4j.sourceforge.net/ evaluation of the bias in the management of patient’s appointments in a pediatric office 15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 8: extract from the prepared data and formatted to decimal values d a t em o n t h d a t ed a y d a t ess n d a t esc h d l n -a pp -f in is h n -o sa -f in is h n -a pp -s u sp n -o sa -s u sp 0.04 0.11 0.02 0.03 0.00 0.01 0.02 0.04 0.05 0.17 0.03 0.01 0.11 0.18 0.07 0.08 0.07 0.16 0.03 0.03 0.04 0.01 0.00 0.01 0.09 0.31 0.04 0.02 0.02 0.01 0.01 0.00 0.11 0.01 0.01 0.01 0.09 0.05 0.00 0.04 after several runs, the "species" with the best suitability value (i.e., the ann topology that generates the least error in the prediction) is obtained in "time" (or cycle) 939 of the 3rd run with a suitability value of 0.0569. the topology thus generated is shown in figure 3. this topology contains: – 4 neurons in the entrance layer. – 4 neurons in the first hidden layer. – 4 neurons in the second hidden layer. – 5 neurons in the third hidden layer. – 3 neurons in the fourth hidden layer. – 9 neurons in the fifth hidden layer. – 4 neurons in the exit layer. figure 3: artificial neural network topology paradigmplus (2020) 1:1 16 cinthia vegega et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . from this defined topology, the ann prototype is constructed using the neurophstudio tool3. then, a new project is generated in which a multi-perceptron backpropagation ann is defined that applies the topology defined by neat4j as can be seen in figure 4. this ann is then trained and validated with the data analyzed in the previous section. although in a real project it would not make sense to validate an intelligent system with the same data with 3http://neuroph.sourceforge.net/ figure 4: ann architecture implemented in neurophstudio paradigmplus (2020) 1:1 http://neuroph.sourceforge.net/ evaluation of the bias in the management of patient’s appointments in a pediatric office 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . which it was trained (given that the precision thus obtained is not reliable), here we only seek to confirm that the probabilities provided by the network can be considered as representative of the data used. in this case, an accuracy of 98.8% is obtained for completed normal appointments, 98.9% for suspended normal appointments, 95.3% for completed over-shift appointments and 97.8% for suspended over-shift appointments. based on these results, it is possible to ensure that this ann is useful to be used as a basis for the intelligent system that predicts the number of normal and over-shift appointments in the doctor’s office. 7 conclusions the repertory grid technique is used to generate an objective representation of the mental image that a person has of the elements in a given area, seeking to go beyond the words that the person could convey. in the present work, an adaptation of this technique is implemented to evaluate the data obtained about the medical appointments of a pediatric office, in order to determine if they can be used for the construction of an intelligent system that predicts the number of normal and overshift appointments in a certain date. from the analysis made, it has been detected that the version of the data is considered sufficiently representative and therefore useful for the implementation of the intelligent system. in turn, this has been confirmed with a prototype with an artificial neural network modeled and trained from these data. as a future line of work, the prototype will be applied in the medical office to evaluate its capacity for generalization in new situations. authors’ information – cinthia vegega is an information systems engineer from the universidad tecnológica nacional facultad regional buenos aires and currently is finishing her master in information systems engineering at the same university. she currently works as interim adjunct professor at universidad tecnológica nacional facultad regional buenos aires and universidad tecnológica nacional facultad regional la plata. – pablo pytel is an information systems engineer from the universidad tecnológica nacional facultad regional buenos aires, has a master in software engineering from the universidad politécnica de madrid, and a phd in computer sciences from the universidad nacional de la plata. he currently works as a professor in the at the universidad tecnológica nacional facultad regional buenos aires and universidad nacional de lanús. – maría florencia pollo-cattaneo is an information systems engineer from the universidad tecnológica nacional facultad regional buenos aires, has a master in software engineering from the universidad politécnica de madrid, and a phd in computer sciences from the universidad nacional de la plata. she currently works as a head professor at the universidad tecnológica nacional facultad regional buenos aires and universidad tecnológica nacional facultad regional la plata. she is also gemis group chair. authors’ contributions – cinthia vegega focused on the specifications of the stages of the proposed process and wrote the manuscript. – pablo pytel supervised and guided the proposed process, managed and supervised the validation of the proposed process, and wrote the manuscript. paradigmplus (2020) 1:1 18 cinthia vegega et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . – maría florencia pollo-cattaneo managed and supervised the validation of the proposed process, and wrote the manuscript. competing interests the authors declare that they have no competing interests. funding the research reported in this paper has been totally funded by the research project "prácticas ingenieriles aplicadas para la implementación de sistemas inteligentes basados en machine learning" (uti5103tc) within universidad tecnológica nacional facultad regional buenos aires. availability of data and material gemis-td-2019-09-tr-2019-10-resultadosturnosmedicos. results of clustering procedure on data for medical appointments in a pediatric office. available at https://github.com/inteligenciaartificialutn/paradigm references [1] e. alpaydin, introduction to machine learning. mit press, 2014. [2] t. m. mitchell, machine learning. mcgraw-hill, new york, 1997. [3] r. s. michalski, j. g. carbonell, and t. m. mitchell, machine learning: an artificial intelligence approach. springer, 1983. [4] l. ljung, “perspectives on system identification,” ifac proceedings volumes, vol. 41, no. 2, pp. 7172–7184, 2008. [5] p. r. cohen and e. a. feigenbaum, the handbook of artificial intelligence, vol. 3. butterworthheinemann, 2014. [6] r. garcía martínez, d. pasquini, and m. servente, sistemas inteligentes. buenos aires, argentina: nueva librería, 2003. [7] p. domingos, “a few useful things to know about machine learning,” communications of the acm, vol. 55, no. 10, pp. 78–87, 2012. [8] g. bontempi, s. b. taieb, and y.-a. le borgne, “machine learning strategies for time series forecasting,” in european business intelligence summer school, pp. 62–77, springer verlag heidelberg, 2013. [9] e. a. r. santoyo and j. a. l. gonzález, “comparación de predicción basada en redes neuronales contra métodos estadísticos en pronósticos de ventas,” ingeniería industrial. actualidad y nuevas tendencias, vol. 4, no. 12, pp. 91–105, 2014. [10] j. pérez porto and a. gardey, “bias definition.” available at https://definicion.de/sesgo/, 2010. accessed: 2020-01-01. [11] m. vivanco, muestreo estadístico. diseño y aplicaciones. santiago de chile, chile: editorial universitaria, 2005. paradigmplus (2020) 1:1 https://github.com/inteligenciaartificialutn/paradigm https://definicion.de/sesgo/ evaluation of the bias in the management of patient’s appointments in a pediatric office 19 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [12] n. collins, “artificial intelligence will be as biased and prejudiced as its human creators,” pacific standard, vol. 1, 2016. [13] k. crawford, “artificial intelligence’s white guy problem,” the new york times, vol. 25, 2016. [14] c. vegega, p. pytel, and m. f. pollo, “método basado en el emparrillado para evaluar los datos aplicables para entrenar algoritmos de aprendizaje automático,” desarrollo e innovación en ingeniería, pp. 106–137, 2017. [15] c. vegega, p. pytel, l. straccia, and m. f. pollo-cattaneo, “evaluation of the bias of student performance data with assistance of expert teacher,” in international conference on applied informatics, pp. 16–31, springer, 2018. [16] s. walczak, “an empirical analysis of data requirements for financial forecasting with neural networks,” journal of management information systems, vol. 17, no. 4, pp. 203–222, 2001. [17] r. j. hyndman, a. v. kostenko, et al., “minimum sample size requirements for seasonal forecasting models,” foresight, vol. 6, no. spring, pp. 12–15, 2007. [18] s. j. raudys and a. k. jain, “small sample size effects in statistical pattern recognition: recommendations for practitioners,” ieee transactions on pattern analysis & machine intelligence, no. 3, pp. 252–264, 1991. [19] d. r. stockwell and a. t. peterson, “effects of sample size on accuracy of species distribution models,” ecological modelling, vol. 148, no. 1, pp. 1–13, 2002. [20] e. alpaydin, machine learning: the new ai. mit press, 2016. [21] t. m. mitchell, the need for biases in learning generalizations. department of computer science, laboratory for computer science research, 1980. [22] p. domingos, the master algorithm: how the quest for the ultimate learning machine will remake our world. basic books, 2015. [23] w. i. limited, “beauty ai. first beauty contest judged by robots.” available at http://beauty. ai/, 2015. accessed: 2020-01-01. [24] s. levin, “beauty contest was judged by ai and the robots didn’t like dark skin,” 2016. the guardian. [25] w. l. perry, predictive policing: the role of crime forecasting in law enforcement operations. rand corporation, 2013. [26] d. robinson and l. koepke, “stuck in a pattern: early evidence on ‘predictive policing’and civil rights,” upturn report, pp. 1–29, 2016. [27] e. edwards, “predictive policing software is more accurate at predicting policing than predicting crime,” huffpost post, 2016. [28] r. lloyd, “critics say a predictive policing system could amplify racial bias in oakland.” https://oaklandnorth.net/2016/11/07/ critics-say-a-predictive-policing-system-could-further-racial-bias-in-oakland/, 2016. [29] northpointe, “practitioner’s guide to compas core.” https://www.equivant.com/wp-content/ uploads/practitioners-guide-to-compas-core-040419.pdf, 2015. paradigmplus (2020) 1:1 http://beauty.ai/ http://beauty.ai/ https://oaklandnorth.net/2016/11/07/critics-say-a-predictive-policing-system-could-further-racial-bias-in-oakland/ https://oaklandnorth.net/2016/11/07/critics-say-a-predictive-policing-system-could-further-racial-bias-in-oakland/ https://www.equivant.com/wp-content/uploads/practitioners-guide-to-compas-core-040419.pdf https://www.equivant.com/wp-content/uploads/practitioners-guide-to-compas-core-040419.pdf 20 cinthia vegega et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [30] j. angwin, j. larson, j. mattu, and l. kirchner, “there’s software used across the country to predict future criminals. and it’s biased against blacks.” https://www.benton.org/headlines/ machine-bias-theres-software-used-across-country-predict-future-criminals-and, 2016. [31] j. trujillano, j. march, and a. sorribas, “aproximación metodológica al uso de redes neuronales artificiales para la predicción de resultados en medicina.,” medicina clínica, vol. 122, pp. 59–67, 2004. [32] g. kelly, the psychology of personal construct. new york, norton, 1955. [33] rae, “real académica española constructo.” [34] f. fransella, r. bell, and d. bannister, a manual for repertory grid technique. john wiley & sons, 2004. [35] r. wolf and h. s. delugach, “knowledge acquisition via tracked repertory grids,” computer science dept., univ. alabama in huntsville, 1996. [36] d. carrizo moreno, “comparación de efectividad de las técnicas de educción de requisitos software: visión novel y experta,” ingeniare. revista chilena de ingeniería, vol. 20, no. 3, pp. 386–397, 2012. [37] m. easterby-smith, “the design, analysis and interpretation of repertory grids,” international journal of man-machine studies, vol. 13, no. 1, pp. 3–24, 1980. [38] p. britos, b. rossi, and r. garcía martínez, “notas sobre didáctica de las etapas de formalización y análisis de resultados de la técnica de emparrillado. un ejemplo,” in proceedings del v congreso internacional de ingeniería informática, pp. 200–209, 1999. [39] p. e. black, “manhattan distance dictionary of algorithms and data structures.” available at https://xlinux.nist.gov/dads//html/manhattandistance.html, 2006. accessed: 2020-0101. [40] j. m. bradshaw, k. m. ford, j. r. adams-webber, and j. h. boose, “beyond the repertory grid: new approaches to constructivist knowledge acquisition tool development,” international journal of intelligent systems, vol. 8, no. 2, pp. 287–333, 1993. [41] l. a. saúl, m. a. lópez-gonzález, a. moreno-pulido, s. corbella, v. compan, and g. feixas, “bibliometric review of the repertory grid technique: 1998–2007,” journal of constructivist psychology, vol. 25, no. 2, pp. 112–131, 2012. [42] p. mcgeorge and g. rugg, “the uses of contrived knowledge elicitation techniques,” expert systems, vol. 9, no. 3, pp. 149–154, 1992. [43] s. stumpf and j. mcdonnell, “using repertory grids to test data quality and expertsh́unches,” in 14th international workshop on database and expert systems applications, 2003. proceedings., pp. 806– 810, ieee, 2003. [44] s. stumpf and j. mcdonnell, “data, information and knowledge quality in retail security decision making,” in proceedings of i-know, vol. 3, pp. 2–4, citeseer, 2003. [45] r. rakotomalala, “tanagra: data mining software for academic and research purposes,” actes de egc, pp. 697–702, 2005. [46] d. e. rumelhart, g. e. hinton, and r. j. williams, “learning representations by backpropagating errors,” parallel distributed processing, vol. 1, 1986. paradigmplus (2020) 1:1 https://www.benton.org/headlines/machine-bias-theres-software-used-across-country-predict-future-criminals-and https://www.benton.org/headlines/machine-bias-theres-software-used-across-country-predict-future-criminals-and https://xlinux.nist.gov/dads//html/manhattandistance.html evaluation of the bias in the management of patient’s appointments in a pediatric office 21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [47] h.-n. robert et al., “theory of the backpropagation neural network,” proc. 1989 ieee ijcnn, vol. 1, pp. 593–605, 1989. [48] h. heidenreich, “neat: an awesome approach to neuroevolution.” https:// towardsdatascience.com/neat-an-awesome-approach-to-neuroevolution-3eca5cc7930f? gi=fb4832b83f34, 2019. accessed: 2020-01-01. paradigmplus (2020) 1:1 https://towardsdatascience.com/neat-an-awesome-approach-to-neuroevolution-3eca5cc7930f?gi=fb4832b83f34 https://towardsdatascience.com/neat-an-awesome-approach-to-neuroevolution-3eca5cc7930f?gi=fb4832b83f34 https://towardsdatascience.com/neat-an-awesome-approach-to-neuroevolution-3eca5cc7930f?gi=fb4832b83f34 introduction problem identification repertory grid context case study and collect data description results of the application of the repertory grid method implementation of the intelligent system prototype conclusions vol. 2, no. 3, pp 14 36, 2021 doi: 10.55969/paradigmplus.v2n3a2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . establishing best pedestrian paths considering sars-cov-2 contagions: mathematical optimization model and mobile application approach juan e. cantor id 1,�, germán a. montoya id 1, and carlos lozano-garzon id 1 1universidad de los andes, bogota, colombia {je.cantor, ga.montoya44, calozanog}@uniandes.edu.co abstract given the modern sars-cov-2 coronavirus pandemic and the universidad de los andes needs to protect their students against possible contagions at the university campus, we established and design a mobile application to obtain the best route between two places in the campus university for a student. the resulting path obtained by our proposed solution reduces the distance traveled by the student as well as a possible coronavirus contagion during his journey through the university campus. therefore, we modeled two types of costs: a transport cost that models the distance to travel the campus by a student, and the contagion risk cost that models the contagion susceptibility that a student has during a displacement through the campus. as a result, we developed and validated a solution algorithm that minimizes the two modeled costs. the algorithm results were compared against a multi-objective mathematical optimization solution were findings show an solution approximation between the algorithm and the mathematical model. finally, a mobile application was developed to map the optimal routes to displace between two points in the university campus given the solution algorithm. keywords: covid-19 ⋅ graph ⋅ pedestrian traffic ⋅ transport function ⋅ interaction function ⋅ bi-objective problem ⋅ heuristic ⋅ mobile application received: 1 november 2021 ⋅ accepted: 12 december 2021 ⋅ published: 24 december 2021. 1 introduction the sars-cov-2 virus pandemic, commonly called covid-19, led countries worldwide to enforce social restrictions to avoid the coronavirus spread. the most common contingency measures, taken by multiple governments, is to limit the citizens traveling, as well as social interactions that occur between them. the previous described measure is commonly known as “social distancing” and has the purpose to limit as much as possible the exchange of segregations between people’s interaction which is the main event that leads to the exponential growth of the covid-19 propagation. [1] however, there are scenarios in which multiple pedestrians spend daily hours walking into multiple agglomerations, where the social distancing measure is not present. in 2019, the universidad de los andes reported around 14 thousand undergraduate students, 3.000 master’s students, and about 364 doctoral students. [2] in a campus area that covers around 11.4 hectares [3]. as it is observed, this article is distributed under the terms of the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. https://doi.org/10.55969/paradigmplus.v2n3a2 https://orcid.org/0000-0003-3185-1986 https://orcid.org/0000-0003-0841-2184 https://orcid.org/0000-0003-2920-6320 establishing best pedestrian paths considering sars-cov-2 contagions 15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . the university institution has a large volume of pedestrians circulating daily through the different facilities of the campus (multiple corridors and interconnection building routes). this scenario was the common behavior before the implementation of virtual classes by the institution due to the pandemic. therefore, given the importance to implement the social distancing measure within the campus considering students agglomerations, we proceed to propose a mathematical model to minimize a student exposure from a covid-19 contagion with also taking into account the shortest distance criteria to be traveled between the two places that a student has to travel across the university campus. the mathematical model was designed based on the characteristic of the solution, which is the modeling of two main objectives based on the problems criteria to be minimized: traveling distance and covid-19 contagion risk between two points on the university campus. therefore, the developed mathematical model was designed for a bi-objective optimization. however, based on the student pedestrian mobility patterns modeled for the corresponding cost parameters related to the traveling distance criteria, the model converged into a mono-objective optimization, given the fact that the modeling resulting into the two main criteria overlapping each other in their optimal solutions. therefore, the approached heuristic which was chosen to be the dijkstra algorithm (as it corresponds to a shortest cost path exact heuristic) only solve one of the two main criteria in each executed scenario. consecutively, the heuristic was validated with the modeled bi-objective mathematical model given a set of scenarios that took into account the criteria to be minimized and a particular source and destiny university building, day and schedule. finally, the heuristic is proposed to be executed and used through a mobile application, where a student pedestrian determines a route to travel the campus in which the contagion risk is minimized at a certain time. additionally, for readers context, it should be noted that the presented research is an extended version of the published version made on the 2021 international conference on applied informatics [4]. this extended version provides more details related to the mathematical model parameters set up -such as data processing, pedestrian traffic displacement pattern and hierarchy data persistence used for the modeling and general overview and future work section. the research content is structured as follows: section 2 describes related works, and the identification and relevance of the problem to be solved. section 3 describes the overall solution design processes made for the final product, which are the definition of the mathematical model and the problems functions modeling (which are the distance to be travel and the contagion risk). section 4 shows the model design process of the multi-objective mathematical optimization with the descriptions and justifications developed for the implementation of the mobile application. section 5 describes the modeling and in-depth description of the problem modeled in the phases presented in section 4. additionally, the section describes the development and validation of the solution algorithm. section 6 shows the mobile application design and development that corresponds to the userfacing solution. section 7 describes the final results obtained by our work. section 8 describes the performance results obtained in the initial iteration of the project, enunciating the main limitations and therefore improvements to be made to the initial iteration. finally, section 9 describes the main tasks and features that have been identified in this research to be done in order to improve software performance and covid-19 risk contagion modeling as a second iteration of the project. 2 related work the following studies emphasized how mobility events are related to the spread of the sars-cov-2 virus. the same covid-19 susceptibility probability equation for an individual in pedestrian congestion was used through these related works. these studies describe simulations to represent the pedestrian mobility patterns under different types of events in a common day. in [5], researchers performed simulations of human mobility models to study the dynamics of coronavirus infection in order to get a dynamic model of the virus spread. to obtain the nature of the epidemiological dynamics, the study made use of the sir (susceptible, infected and recovered) paradigmplus (2021) 2:3 16 juan e. cantor et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . model. through contagion rates planned in the model, the behavior of different groups populations was simulated. if a susceptible person and an infected person meet, there is a probability that the susceptible person will become infected. time after infection,the person typically recovers. therefore, the study introduced a contact graph constructed from everyday situations such as agglomerations in public trans-port, work, home, and others (scenarios in which the sir model could be applied and studied). finally, after executing the simulations of the contact graph with the epidemiological dynamics previously stated, the study set out strategies as conclusions, such as the suspension of public transport, total quarantine, and others social decisions, in order to prevent and reduce the spread of the virus. for instance, one of the researched works created an epidemiological model based on agents (called pancitysim) in which the movement of many individuals were modeled in a city. for this purpose, the model used a graph to represent the contact of many activities performed by the population in the whole city. in this sense, the spatio-temporal properties of an epidemic were studied according to the sir model in order to represent the impact of human interactions to the transmission of the virus [6]. thus, it was feasible to study human mobility models in cities in order to understand the covid19 spreading behavior. based on these related works, a solution could be applied to the mobility patterns associated with pedestrians and their environment within the universidad de los andes, as well as mathematical models and probabilities associated with the spreading of the coronavirus. additionally, as the date of the research development, there was no evidence of work of the covid-19 contagion risk optimization applied to a specific pedestrian traffic scenario, and therefore, an optimal solution application that a user could use during their traveling (based on a scenario where the users faces pedestrian agglomerations that increase the coronavirus contagion probability). hence, based on the existing human mobility models simulations, we find the opportunity to use as base the epidemiological models to optimize pedestrian traffic from a concrete scenario and related data, and also to expose the solution through a software tool that could be used in real-time. 3 general solution design the design process established for the solution implementation is described below. figure 1 shows a diagram of the main activities developed in the process, as well as the tools, investigations and results associated with each activity. in the diagram each box (white) has the statement of each activity written, where their entries (green) and the respective software tool (blue) or associated research (yellow) are shown. the upper left numbers represent the sequence step when the activity was develop. finally, the results or outputs of each activity (red) and the arrows indicate the dependencies between activities and inputs. as a reference for details of each main activity, section 4 describes the detail of the activities definition of initial mathematical model, interaction cost function, pedestrian traffic simulation on university campus and pareto front development. following is an general overview of the inputs, outputs and the description of each main activity 3.1 definition of initial mathematical model first, it was modeled through the geojson website an graph by plotting points on the university map where there was any routes across the university campus. then, based on a dataset with anonymous student data related to the classes schedule that the students have, it was generated random pedestrian based on the potential nodes that a student has to travel in order to attend to their buildings’ classes. the mobility patterns and the loading of the pedestrian traffic was implemented through python scripts and libraries. paradigmplus (2021) 2:3 establishing best pedestrian paths considering sars-cov-2 contagions 17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 1: diagram showing the inputs and outputs of each activity executed in the design process. finally, as activity output, we developed the design and modeling of f1 function, which is the function that represents the distance (measured in units) between each pair of nodes that have a link on the university. the modeling activity includes a sets, parameters and variables definition for the f1 function. 3.2 interaction cost function for the construction of the f2 function, with is the function that represents the covid-19 probability risk contagion between each pair of nodes that have a link on the university, we used the proposed equation in [5], which models the susceptibility probability of contracting the coronavirus based on the numbers and characteristics of the social encounters that people have during their displacements. 3.3 pedestrian traffic simulation on university campus based the culmination of the two previous activities, it was proceeded to model the student pedestrian traffic, which assign the respective costs to f1 and f2 on each link of each pair of nodes that have a connection on the university graph. as inputs, we considered the pre-processed data of the student classes schedule dataset (which was done through the software tool knime) to generate mobility patterns and displacement that students would have and consecutively the assignation of the social encounters (costs for the interaction cost function) between each existing link on the university graph. the interaction costs between each pair of nodes with connection was denoted as cij, where it is represented the amount of students a student would encounter when traveling from node i to node j also, to determine the initial nodes where students would began their displacement, we took the [7] as a reference where the students would enter to the university in terms of medium transport arrival nodes. finally, we decided to store all the loaded students mobility patterns data on the firebase’ firestore non-relational database, which allows to store more efficient this type of data based on the hierarchy presented (student pedestrian traffic based on a academic cycle, a day and a certain schedule). paradigmplus (2021) 2:3 18 juan e. cantor et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 pareto front development based on the setup of the two functions (distance and interaction costs functions), we implemented a pareto front taking into account the modeled functions. 4 multi-objective mathematical optimization model we establish a mathematical model to minimize the distance traveled by a student and the covid-19 contagion risk when facing others students during the displacement. therefore, the mathematical model included a transportation cost matrix represented as the distances between each pair of nodes included in the route of the map to travel. likewise, an interaction costs matrix was also established, where it is represented the student pedestrian traffic through the nodes in the university campus map. additionally, in the mathematical model we defined a minimum coverage radius for each node, which is the maximum distance that must be for each pair of nodes to be a connection. we define the initial nodes network using geojson web tool [8], where graph nodes were placed consecutively on each existing university campus path. once the coordinates of the different nodes had been exported in a file in json format, they were loaded as parameters to the mathematical model. for the mathematical model, sets, variables, and parameters understanding, see table 1. table 1: sets, parameters and variables of the initial mathematical model category symbol description sets n number of nodes in the network parameters ciij interaction cost between node i and node j, which is the number of people that walk through the link (i,j). ctij transport cost between node i and node j, which is the euclidean distance between these nodes s source node number, in which a student started its walk d destination node number, in which a student finished its walk rc node’s ratio coverage, which is the maximum distance that two pairs of nodes must have to exist a connection link between them. variables xij binary decision variable that shows whether the student chooses the link between node i and node j as a part of its path. from the source node to the destination node. 0 if otherwise. hence, to validate the proposed model, the following objective function was defined: minimize ∑ iϵn ∑ jϵn xij ⋅(ctij + ciij) (1) subject to: ∑ jϵn xij = 1 ∀iϵn∣i = s (2) ∑ jϵn xij = 0 ∀iϵn∣i = s (3) ∑ iϵn xij = 1 ∀jϵn∣i = d (4) ∑ iϵn xij = 0 ∀jϵn∣i = d (5) paradigmplus (2021) 2:3 establishing best pedestrian paths considering sars-cov-2 contagions 19 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ∑ jϵn xij − ∑ jϵn xji = 0 ∀iϵn∣i ≠ d ∧ i ≠ s (6) xij ⋅ ctij ≤ rc ∀i, jϵn (7) the objective function of the mathematical model is subjected to the constraints represented in equation 2 to equation 6. in equation 2 it is indicated that the source node should only have one outflow, which dictates that a student can only take one path from its origin node. equation 3 ensures that a student cannot returned to its source node. equation 4 ensures that the student reaches its destination node. equation 5 establishes that once a student arrives to the destination node, he cannot travel to other paths or nodes. for intermediate nodes between the source node and the destination node, equation 6 establishes that if a student reaches an intermediate node, he/she has to leave that node. finally, to guarantee that the model chooses links within the ratio coverage of each node, equation 7 was proposed. for the mathematical model implementation, we executed the model on the pyomo optimization framework on an intel ®core i5-6200u, 2.3 ghz, 7.7 gib system. 4.1 interaction cost function the susceptibility probability of contagion proposed by [5] was used in our work, which can be observed in equation 7. pn,t = 1 − e−θ⋅∑mϵm qm,t⋅inm,t⋅tnm,t (8) where m is the set of students with whom a student was in close contact, with the following variables and parameters: pn,t is the probability of a student n to receive an infectious dose. however, as stated in [9], this probability should not be understood as “probability of contracting the virus”, since this depends on the immune system of each student. θ is the calibration factor for a particular disease. qnm,t corresponds to the microbial load between two people, which is the amount of virus that the student m transmits in time t. inm,t is the contact intensity between the person n and m, which corresponds to the distance. finally, tnm,t represents the time that a student n interacts with a student m during an interval t. for the simplicity of the proposed mathematical model, we assumed that the distance between each pedestrian and each user corresponds to 1 meter and that their interaction time was 0.5 seconds. however, due to the fact that as of the date this research was developed, no calibration parameters and microbial load of covid-19 were known. thus, parameters estimated by the previously mentioned studies were used. first, the value qm,t was assigned as 1 based on estimates made by [5]. similarly, the calibration parameter θ was taken as 120 based on studies conducted by [9] in closed spaces such as aircraft interiors during commercial flights. we considered that the choice of this calibration value should be made on the assumption that all routes in the campus rely on indoors spaces (which is the assumption of a pessimistic scenario). with this decision, it was possible to establish an equal load of micro-organisms in the environment for all the connections between nodes, which allowed the model to calculate the best path based on the segregations caused by other students and not on the dispersion caused by other external factors. 4.2 pedestrian traffic simulation on the university campus 4.2.1 data preprocessing to establish the parameters linked to the interaction cost of the model, students pedestrian traffic was simulated in a post-covid-19 scenario, which is a scenario in which the traffic of students within the paradigmplus (2021) 2:3 20 juan e. cantor et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . university campus is not affected by the current pandemic at the date of publishing this research. for this reason, the anonymized schedules of undergraduate students in the first semester of 2019 dataset was processed, as well as the semester current academic schedule. then, the data was pre-processed and cleaned through the data analysis software knime [10]. using this tool, the list of courses was matched to the courses registered by the students. the operations performed on the data set are presented below: – registered courses that did not have a classroom attribute were deleted. since these courses had virtual modality, the students enrolled in those courses did not generate pedestrian traffic. – registered courses that did not have a record of start or end time were deleted. without having a specific range of time the traffic generated by students in a period of time on a given day is not known. – courses attribute values such as space and availability were deleted, as they were not significant for the pedestrian traffic simulation. – for simplicity of the database entries, the secondary classrooms of the enrolled courses were eliminated, taking as pedestrian traffic the displacement towards the primary classrooms of their registered courses. – the rows that contained rooms outside the main campus (i.e. practice center) were eliminated since the network of nodes to be modeled only have into account buildings that are located within the main campus of the university. finally, the result was a dataset that had the following attributes with a total of 65, 536 records. the final courses attributes are shown in table 2 with their respective description. table 2: fields of the trimmed dataset used to generate pedestrian traffic. data field name (written in spanish) description id student id. nrc course id. materia department or faculty of the course. nombrecurso name of the course. parte current academic cycle of the course. salon1 building where the course was taught. hini1 time and minutes when the course started. fini1 time and minutes when course ended. l,m,i,j,v,s days on which the course was taught. 4.2.2 configuration and loading of pedestrians’ initial and intermediate nodes once the final dataset to be used as parameters in the model was obtained, the students pedestrian traffic data was loaded into a non-relational database. for this task, a firestore database was created on the platform firebase [11], and for the loading data process, scripts written in python programming language were used [12]. then, each dataset record was uploaded as the origin and destination buildings that each student has in a change of classes in an academic cycle, day and specific time. the following describes how special cases were handled when the record consisted of the first class of the day, or when a student had two or more non-consecutive classes on the same day: paradigmplus (2021) 2:3 establishing best pedestrian paths considering sars-cov-2 contagions 21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . – for a student’s first class of the day, it was decided that a student would arrive at the university campus through the transport nodes identified in uniandina mobility survey (encuesta de movilidad uniandina) carried out in may 2019 [7] , where each means of transport has an associated percentage, which represents the percentage of students that travels on the respective means of transport. then, a random variable that oscillated within the ranges of percentages was defined, and from that variable value, it was assigned within the closest mean of transport percentage. likewise, the network nodes equivalent to where the transportation stops are located within the university campus were defined. modes of transportation such as hike were omitted, since it does not exist or is not known with certainty. also it was considered that students can arrive from multiple points in the network using this type of modes. – for non-consecutive classes belonging to the same day, it was decided to define random nodes where students would spend their free hours. the choice of these nodes was made arbitrarily. the set of nodes assigned in free hours consisted of the prado el bobo, cafeteria central and biblioteca ramón de zubiría (these places are part of the university campus). once the data had been uploaded to the database, a hierarchical non-relational database was obtained based on the academic year, day, and class change schedule. in figure 2 you can see a diagram of the database schema. for the structure hierarchy of the database, it was composed from top to bottom as follows: – academic cycles: semester period when the classes are taught. this allows to characterize the student pedestrian traffic based on the current period. the selection of a particular academic cycles helps to filter pedestrian congestion peaks based on a certain period of time. examples of academic cycles values includes first and last 8 semester week (8a and 8b respectively). – classes days: laboral weekdays. this allows to segregate pedestrian traffic based on the classes days that students have. laboral weekdays go from monday to saturday. figure 2: diagram that shows the hierarchy of data uploaded to the non-relational database. paradigmplus (2021) 2:3 22 juan e. cantor et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . – change of class schedule: intervals between classes which are the times where student make their displacements from the source node (which could be transport medium node or a building node where they were having a previous class) to their new destiny node. the modeling of this interval is dictated by how the university schedules the intervals of the courses. the rule is applied in general for all classes scheduled in the university, where all classes start at o’clock or half hour pass o’clock, and end 10 minutes before those times. intervals go from the first interval for the first class of the day (6:30 a.m.) to the last class taught on a day (6:30 p.m.) – student id: student dataset identification. students contained in the change of classes interval set, where it is indicated that a student would displace during said interval. – source/destiny place or building of each student: source and destiny name where the student will travel across the university campus. with the source/destiny of multiple pedestrians as the last bottom level, pedestrians data could be effectively loaded to the parameters of the interaction costs of the mathematical model and heuristic, as it only takes for parameters the source and destiny of a list of multiple student pedestrians. 4.2.3 pedestrian traffic’ displacement pattern for modeling the route pattern that each student was going to travel when moving around the campus, we decided that for the simulation the students would seek for the shortest way to displace from their source building to their destination building. for this task, it was assumed that each student would move through their route nodes using the dijkstra algorithm for minimal cost path (taking the distance between each pair of nodes as the cost) [13]. consequently, the selection of each path by each student set the interaction costs of the mathematical model and the solution heuristic which is described in depth in section 5 -. at table 3 is the terminology used for algorithm 1, which was used to model the pedestrian traffic parameters to the interaction cost variable. as a general description of the algorithm, the algorithm seeks to assign to every student the set of links that the student would travel as a mobility pattern for going from their source node to their destiny node, and consecutively assign for each link that the student will travel the aggregated interaction cost for that link in a global variable. in the algorithm’s first loop, it is assign the optimal set of links (given by dijkstra algorithm) that each student in the pedestrian traffic will take. then, in the second loop it is checked for all pair of links in the university graph if the link is associated with a link choose by the dijkstra algorithm for a students’ path, then the counter of the number of students traveling through that link is increased (which is the interaction cost related to that specific link). algorithm 1 algorithm for the simulation of pedestrian traffic and interaction costs assignment to the network nodes’ links cij = 0 ∀i, jϵn sp(e)={} ∀e ϵ e for all e ine do sp(e) = dijkstra(s(e), d(e)) for all i, j in n do if i, j in sp(e) then cij ← cij + 1 end if end for end for paradigmplus (2021) 2:3 establishing best pedestrian paths considering sars-cov-2 contagions 23 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 3: related terminology used for the algorithm 1 symbol description assumptions n set of network nodes. e set of students that displace in a specific academic cycle, day and schedule. s(i) student source node i. ∀iϵe ∀s(i)ϵn d(i) student destiny node i. ∀iϵe ∀d(i)ϵn sp(i) optimal set of links that a student i has to take to displace from s(i) to d(i). ∀iϵe ∀s(i), d(i)ϵn dij optimal links calculated by dijkstra algorithm from node i to node j displacement. ∀i, jϵn cij interaction cost, which the amount of pedestrian from node i to node j during a certain period of time. ∀i, jϵn 4.3 pareto front development consecutively, we implemented a pareto front to observe the behavior of the transport and the interaction cost functions when they are minimized and one function has more weight over the other. therefore, the pareto front was executed through the method ε -constraint [14]. for this calculation, the previously defined network of nodes was used and the parameters of the transport function were assigned based on the traffic loaded in the database. finally, we defined a weights vector in which each vector’ element multiply each function on each iteration, in order to give greater weight to one function compared to the other. the mathematical expression was defined: w = [10, 20, . . . , 50] (9) where for each element of w would represent the upper limit that f2 should have when wanting to minimize f2 (this is reflected in the additional restriction described below). therefore, each element w was iterated to obtain values close to the limit of f2. it should be noted that the mathematical model of the method also includes the restrictions initially set in the section 4. the pareto front execution was computed on an intel ®core i5-6200u, 2.3 ghz, 7.7 gib. for each wϵw : f1 = ∑ iϵn ∑ jϵn xij ⋅ cij (10) f2 = ∑ iϵn ∑ jϵn xij ⋅ cij (11) minimize f1 (12) subject to: f2 ≤ w (13) see equation 2 equation 7 5 heuristic as the research problem can be represented as a minimum cost problem, where for each node’s link exists two types of cost transport cost and interaction cost we used a known heuristic to solve the paradigmplus (2021) 2:3 24 juan e. cantor et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . minimum cost problem by combining in a same scale the two types of cost for each link. the dijkstra algorithm [13] was used as a solution algorithm, where the student through the developed mobile application can choose the criterion to minimize given a set of parameters. regarding the costs unification, the costs for each link were normalized, and then multiplied by a weight represented by a scalar between 1 and 0. finally, the product of the normalized costs and their scalar was added, resulting in a single cost matrix. it should be noted that the scalar weight is the representation of the path criterion selected by the user (i.e. 0 as least importance, 1 as greatest importance). however, we found that given the modeling of pedestrian traffic described in the sub-section 4.2 and the obtained pareto front results in different scenarios, the problem of minimization of the transport cost function and the interaction cost function only had solutions in the limits of each function, which is a solution was not found that minimized the two objectives "as equal", but rather two optimal solutions were obtained that minimize only one function. 5.1 bi-objective problem understanding to understand the reason for the existence of only two optimal solutions for the minimization problem of f1 and f2 (as conventions, f1 is referred as the transportation cost function and f2 is referred as the interaction cost function.), it must be remembered how the distribution of traffic was defined in the sub-section 4.2, which was done under the assumption that every student who travel through the campus would choose the shortest route in the network of nodes. scenarios were obtained where all the costs associated with f2 are distributed by the links that minimize f1. therefore, when it is desired to minimize only f1, links are obtained when it is desired to maximize f2. similarly, when only f2 is minimized, the chosen links have no costs for this function. those links, since they are not optimal for f1, maximize the value of said function. to describe the understanding of the problem in more detail, see figure 3. in the diagram case, it it observed that in each optimal link chosen by dijkstra’s algorithm (colored with blue and where dϵd) between a source (colored with green and labeled with an "s") and destiny (colored with red and labeled with an "d") node have all the f2 associated costs distributed, which generates only two options to minimize each function. figure 3: example of a graph in which each k link has two types of cost: (f k1 , f k2 ). paradigmplus (2021) 2:3 establishing best pedestrian paths considering sars-cov-2 contagions 25 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . for the formal explaining of the problem nature, see the enunciated terminology described below: be d ∶ set of optimal links selected by dijkstra’s algorithm c1 ∶ set of optimal links to minimizef1 c2 ∶ set of links where there are costs associated withf2 where from the pedestrian traffic displacement model, the following sets relationships occurs: c1ϵd c2ϵd 5.2 single cost approach to solution heuristic for the solution heuristic we proceeded to use dijkstra algorithm with a unique cost calculated from the weighting of the normalization of the f1 and f2 costs. for this, the following scale characteristic normalization method was applied: f ′ 1(i, j)= f1(i, j)− min(f1) max(f1)− min(f1) (14) f ′ 2(i, j)= f2(i, j)− min(f2) max(f2)− min(f2) (15) where f ′k(i, j) is the normalization of the scaling characteristic of fk in the link of node i and node j. max(fk) corresponds to the maximum value in the data set of the function k. for the case of both functions, the maximum value was the integer 99999, since that was the default value in the cost matrix where there was no link between a pair of nodes. finally, max(fk) is the minimum value in the data set of the function k. for f1 it corresponded to the link between nodes of shorter length, and for f2 it was the minimum number of students who traveled on a link given a cycle, day and time. finally, the costs of the normalized functions were weighted through the weights that were later defined from the input parameters entered by the user, and they were assigned to a new function. likewise, from the new function, we defined the heuristic solution that would have as parameters the inputs received by the front-end of the mobile application: f(i, j)= w ⋅ f ′ 1(i, j)+(1 − w) ⋅ f ′ 2(i, j) (16) h(s, d)= dijkstra(f, s, d) (17) where f is the function that stores the weight of the two initial normalized functions. w corresponds to the weight that the user implicitly enters as a parameter in the front-end to give a weight to each function. dijkstra is the function that returns the optimal links given the unified cost function f , a source node s and a destination node d given a cycle, day and time. finally, h is the function that represents the problem’s heuristic solution. once the heuristic solution was modeled, the results of the algorithm were compared with the solutions provided by the pareto front in different scenarios. 5.3 results comparison and validation with the pareto front for the validation of the results produced by the heuristic developed with the pareto front points, we defined different scenarios to vary the pedestrian traffic given a cycle, day and time with the same origin and destination node in each scenario. this was made with the objective of comparing how is the variation of the optimal route of two points from the assignment of different costs of f2 (since the costs of f1, being the length of each link between nodes remained fixed in the different executions. however, it should be noted that the accumulated cost of f1 changes from the different routes selected). paradigmplus (2021) 2:3 26 juan e. cantor et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 4: specifications of the scenarios chosen for the comparison of the solutions thrown by the solution heuristic and the pareto front. scenario number academic cycle day schedule source building destiny building 1 1 wednesday 10:50 11:00 j. m. santodomingo mario laserna 2 1 monday 13:30 14:00 j. m. santodomingo mario laserna 3 1 thursday 13:30 14:00 j. m. santodomingo mario laserna as it is observed in table 4, the scenario specifications include the chosen academic cycle (semester period), day, schedule (interval of time where student pedestrian displace through the university campus) source and destiny building. as it observed, the source and destiny building remain fixed through all the three scenarios, where the days and schedule are variable. this is done with the purpose of having the same source and destiny for all scenarios, varying the time parameters that change pedestrian traffic peaks and volumes. for the generation of the pareto front, the method described in sub-section 4.3 was used. regarding the values of w for f , different values were iterated in the domain of w ([0, ..., 1]) to observe the obtained points by the heuristic in each iteration. it is observed in figure 4 that different points were got from the pareto front, as well as the heuristic solution from the same source and destiny buildings, but with different student pedestrian traffic and assigned weights for each wϵf . for the pareto front and heuristic solutions obtained for scenario 1 (subfigure a), it is observed that given the pedestrian traffic for the 10:50-11:00 schedule on wednesday, it was possible to obtain more intermediate solutions given the concavity of the curve compared to the other scenario fronts. for scenario 2 and scenario 3 (subfigures b and c respectively), the optimal solutions given by the mathematical model were more approximate to the optimal solution that only solve for either f1 or f2 functions. also, for this two scenarios, it was found that the solution given by the heuristic when minimizing only f2 was not included in any of the pareto front resulted in the executed mathematical model solving. as main evidence, we observed that since it is a problem with only two optimal solutions available the pareto front, points were generated particularly at the limits of each function. likewise, it was obtained that when w ≠ 0 or w ≠ 1, the heuristic yielded the same solution in each iteration. therefore, the solutions thrown by the heuristics behave as if they were only minimize f2, this is due to a single cost function where the costs of f2 associated with the optimal f1 paths made the heuristic look for paths that were not optimal for f1 in order to minimize weight associated with f2. for such reason it can be seen that the algorithm solution does not generate optimal results to minimize f1. thus, in the iteration when w = 1which is when the heuristics were only considered minimizing f1a solution was got at one end of the pareto front, where the associated cost of f1 is minimized and the cost associated with f2 is maximized. however, on iteration when w = 0, which is the case where the heuristic only minimized f2. we found that although that the solutions obtained minimize the accumulated cost of f2 with respect to the iteration scenario when w = 1, these solutions are solutions dominated by the points that make up the front, and additionally, it is a non-optimal solution when minimizing f2 compared to the solutions obtained in the iterations when w ∉{0, 1}for additional information of the described scenarios, see figure 5, in where the optimal path selected by the heuristic is shown for the different values iterated by w in its domain. note the similarity of routes chosen when the heuristic only minimizes f1 or f2 (subfigure a and c respectively) , and as in a different scenario (subfigure b ) increases the cumulative cost of f1 to minimize the cumulative cost of f2. from the observations derived from the results, and from the naturalness of the modeled problem described in sub-section 4.2, we decided to only expose two possibilities of paths in the mobile application, where each option corresponds to the minimization of each function. therefore, to obtain these results, the domain of w was configured to only take the values of 1 and 0.5 (which is a paradigmplus (2021) 2:3 establishing best pedestrian paths considering sars-cov-2 contagions 27 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 0.0 0.2 0.4 0.6 0.8 1.0 f1 costs 0.0 0.2 0.4 0.6 0.8 1.0 f2 c os ts pareto front curve pareto front point heuristic: min(f2) heuristic: min(w * f'1 + (1-w) * f'2) heuristic: min(f1) scenario 1 (a) pareto fronts and heuristic solutions for scenario 1. 0.0 0.2 0.4 0.6 0.8 1.0 f1 costs 0.0 0.2 0.4 0.6 0.8 1.0 f2 c os ts pareto front curve pareto front point heuristic: min(f2) heuristic: min(w * f'1 + (1-w) * f'2) heuristic: min(f1) scenario 2 (b) pareto fronts and heuristic solutions for scenario 2. 0.0 0.2 0.4 0.6 0.8 1.0 f1 costs 0.0 0.2 0.4 0.6 0.8 1.0 f2 c os ts pareto front curve pareto front point heuristic: min(f2) heuristic: min(w * f'1 + (1-w) * f'2) heuristic: min(f1) scenario 3 (c) pareto fronts and heuristic solutions for scenario 3. figure 4: pareto fronts and heuristic solutions for scenarios 1, 2 and 3 decimal that assigns equal weight to both normalized functions). 5.4 results comparisons and validation based on the mathematical model from the configurations established for the solution heuristic in the sub-section 4.2 where f1 and f2 were equally normalized and weighted to minimize f2 the execution times used by both the mathematical model and the heuristics to minimize f1 and f2 were compared, as well as the values established by the functions in each scenario. the mathematical model was configured in the optimization framework pyomo [15] and the heuristic was written and executed in a script written in the programming language python. the specifications of the chosen and executed scenarios were set up as the same scenarios for the pareto front validation (which details can be observed in table 4). first, the execution times of each scenario were compared, where it was noted that the execution times got by the heuristic are approximately 98.45% less than the times obtained by the mathematical model. the previous result facilitates the response time of the calculation of an optimal route when the heuristic is integrated as a service to the mobile application. paradigmplus (2021) 2:3 28 juan e. cantor et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . −0.007 −0.006 −0.005 −0.004 −7.406e1 4.601 4.602 4.603 4.604 4.605 node min(f1) (a) optimal calculated path when f1 is the selected criteria to be minimized. −0.007 −0.006 −0.005 −0.004 −7.406e1 4.601 4.602 4.603 4.604 4.605 node min(0.5 * f'1 + 0.5 * f'2) (b) optimal calculated path when both f1 and f2 are the selected criteria to be minimized. −0.007 −0.006 −0.005 −0.004 −7.406e1 4.601 4.602 4.603 4.604 4.605 node min(f2) (c) optimal calculated path when f2 is the selected criteria to be minimized. figure 5: comparison of calculated optimal routes when choosing different criteria to be minimized both the heuristic and the mathematical model were executed on an intel ®core i5-6200u, 2.3 ghz, 7.7 gib. additionally, in the executions made by pyomo the mathematical model was solved using the glpk solver. the execution times details and values for each scenario can be observed in the table 5, table 6 and table 7. subsequently, the resulting values of f1 and f2 were compared for each tool of the minimization of each function of each scenario. as it is observed in table 6 and table 7, the heuristics got relatively lower values for f2 compared to the mathematical model in each case of minimization of f1 (it should be noted that in these cases the two tools got the same result for f1), while in the cases of minimization of f2 each tool had the same results for this function. 6 mobile application design we implemented a rest api software architecture style [16] using the web framework flask [17] written with python scripts. likewise, the steps and instructions for the heuristic solution (see subsection 5.2) were developed in the programming language python. the input parameters that the paradigmplus (2021) 2:3 establishing best pedestrian paths considering sars-cov-2 contagions 29 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 5: comparison of the execution times of the heuristic and the mathematical model in each scenario. scenario number function to minimize heuristic execution time (sec) mathematical model execution time (sec) 1 f1 2.761 182.504 f2 2.519 164.167 2 f1 2.522 153.907 f2 2.423 171.088 3 f1 2.544 161.058 f2 2.695 165.645 table 6: comparison of the resulting values f1 in the execution of each scenario. scenario number function to minimize heuristic value for f1 (units) mathematical model for f1units 1 f1 3.08 ⋅10−3 3.08 ⋅10−3 f2 6.58 ⋅10−3 6.99 ⋅10−3 2 f1 3.08 ⋅10−3 3.08 ⋅10−3 f2 6.68 ⋅10−3 7.48 ⋅10−3 3 f1 3.08 ⋅10−3 3.08 ⋅10−3 f2 6.64 ⋅10−3 7.11 ⋅10−3 table 7: comparison of the resulting values f2 in the execution of each scenario. scenario number function to minimize heuristic value for f1 (units) mathematical model for f1units 1 f1 495 495 f2 5 5 2 f1 540 540 f2 19 19 3 f1 682 682 f2 35 35 rest api received for the solution heuristic to compute the optimal path were: a string text containing the initials of the building in which the student is located (s); a string text containing the acronym of the building the student wants to go to (d); a string text that shows the academic cycle in which it is desired to calculate pedestrian traffic. by default, this value corresponds to "1" (cycle); a string text that indicates the day set to calculate pedestrian traffic (day); a string text that indicates the schedule for changing classes of pedestrian traffic. note that class change hours are expressed in the 24-hour system (schedule); an integer that shows which function it is wanted minimize (1) or if it is wanted to minimize the transport cost function (0), if it is wanted to minimize the interaction cost function (w). for more details on what these functions represent, see section 4. it should be noted that these methods are received by the server through a post request. to test the connection between the flask server and the client, the server was run on a local machine intel ®core i5-6200u, 2.3 ghz, 7.7 gib machine. and perform client api requests through the postman [18] software. for the mobile application development, we designed it in a way that it could be executed in the operating system android 9 [19]. likewise, through the programming language kotlin [20] and the graphical interfaces for the design of the ui components provided by the android studio development paradigmplus (2021) 2:3 30 juan e. cantor et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . environment, we implemented the front end logic and graphical components to show the probability of susceptibility and the accumulated distance when traveling a route. for the generation of the university campus map, a google maps software development kit was used available for android [21]. in figure 6, it can be observed the graphical interface of the application with the susceptibility probability and the accumulated distance in the lower bar as well as the optimal route drawn. it should be noted that the icons and figures of the application were taken from the web page https://www.flaticon.com/ [22]. for the selection of parameters to be sent to the server, a graphical android component (alertdialog) was used to create a form that would allow a student to select the source building, the destiny building, the opening hours, change of classes available for that day and the route criteria to be minimized. to get significant records of the students’ trajectory for the generation of pedestrian traffic, we assumed that the default academic cycle parameter was 1. likewise, the day was automatically assigned according to the current day on which the application was used. if the student uses the application on sunday the day on which the university does not hold academic classes the application assigns monday as the default parameter. in figure 6, it can be observed the parameter selection form. once the parameters were selected, when pressing the get path button of the form, the application using the http android volley [23] library send the request post to the ip address of the local machine where the server was deployed. once the server response is received, the application updated its interface, showing in the lower bar the susceptibility probability that the user has when traveling (a) route with specific parameters. (b) parameter selection form with the distance minimization criterion selected. (c) parameter selection form with the susceptibility minimization criteria selected. figure 6: screenshots of the mobile application paradigmplus (2021) 2:3 https://www.flaticon.com/ establishing best pedestrian paths considering sars-cov-2 contagions 31 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . the route, as well as the accumulated distance that they have to walk. an example of how the route is drawn in the application can be observed in the figure 6. 6.1 software architecture in summary, once the back end and the front end of the application solution to the initial problem were established, a software architecture was built where the following sequence of steps is performed in order to obtain a route that minimizes one of the two criteria to consider in the problem following is a summary of the general process that the mobile application follows in order to calculate an optimal path: 1○ the user through the mobile application selects the parameters to calculate the most optimal route, among these parameters is the selection of the function to minimize. 2○ the server receives these parameters and proceeds to calculate the transport cost matrix in order to generate pedestrian traffic given the time parameters assigned by the user, and likewise, assigns these costs to the interaction cost matrix for each link of the university graph. the records of the routes of all students are obtained from the platform firebase [11] 3○ once the cost matrices are obtained given the scenario assigned by the user, the solution heuristic is executed for the chosen function and the susceptibility probability, the accumulated distance and the list of coordinates of the nodes of the optimal route. 4○ once the mobile application receives the values from the server, it proceeds to graph those values and path in the graphical interface. in figure 7 it is observed the final architecture diagram. as it is displayed, in the upper right corner of each activity / device it can be observed the programming language, software or platform used. in the upper left corner is the step number of the sequence of steps followed by the process execution. figure 7: diagram of the conceptual software architecture built. paradigmplus (2021) 2:3 32 juan e. cantor et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . the intermediate texts between the arrows connecting each activity / device show the communication protocols used. 6.2 validations to observe the final product functionalities through its graphical interface, the scenarios presented in table 8 were considered and validated to notice the variation between the probability of susceptibility and the accumulated distance when minimizing each criteria, given the different origin-destination buildings and pedestrian traffic on a certain day and class change schedule. these results describe the obtained values on the mobile application ui for each criteria based on the selected criteria to minimize. for the susceptibility probability column, it is shown when choosing the susceptibility criteria in the mobile application the probability is reduced compared to the same scenario when the distance criteria is selected for the path calculation. as it is observed, the variables susceptibility probability and distance are inversely proportional it should be noted that the scenarios presented took thursday and "1" as the default day and academic cycle respectively, since the tests were made on that specific day of the week. table 8: specifications of the tested scenarios on the mobile application. no schedule sourcebuilding destiny building selected criteria s. probability value (percentage) a. distance value (units) 1 10:50 11:00 mario laserna aulas distance 99% 0.0016 susceptibility 85% 0.0025 2 13:30 14:00 mario laserna city u distance 99% 0.0023 susceptibility 66% 0.0037 3 10:50 11:00 j. m. santodomingo mario laserna distance 100% 0.0031 susceptibility 99% 0.0068 7 conclusions based on this research, a mathematical model and solution algorithm were developed in order to optimize the most important criteria to be considered by a universidad de los andes student: displacement effectiveness and covid-19 contagion risk. therefore, the mobile application that was developed allows a student pedestrian to access in a easy and quickly way a software tool that optimizes the covid-19 risk contagion problem from a mobility pattern and displacements perspective. likewise, based on the pedestrian traffic simulation and the modeled cost functions, we could generate and understand a pedestrian agglomerations’ approximation based on a set of student classes that had time attributes. moreover, based on the data hierarchy and aggregated pedestrian mobile users data, and considering the case the developed mobile application is used by multiple users, strategies can be established for data analysis from the different hierarchy data levels (day, academic cycle, etc.). this activity has the purpose to understand the mobility patterns that university students modelate, and therefore, enables high-level decision making processes that avoid pedestrian agglomerations at a specific time. 8 performance and limitations through the research development we found perfomance cases in which from the variables quantity and the initial costs functions modeling cause additional response times in the solution software exeparadigmplus (2021) 2:3 establishing best pedestrian paths considering sars-cov-2 contagions 33 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . cution. following are described the events that had an impact on the final product performance, with considerations to optimize and improve those respective events. 8.1 university campus graph modeling optimization given that the mathematical model costs matrices were nxn size, where n was the number of consecutive nodes established for each existing path on the university campus, and given the node network size of the university campus (which were 700 approx.) execution times for pareto front and mathematical model solving execution were longer than 1 minute. therefore, we consider to reduce the university campus nodes quantity, suppressing innecesary intermediary nodes between the beginning and end of each existing path on the university. with this feature, execution times of algorithms and mathematical models solving that consume the matrices costs would be reduced. 8.2 student pedestrian traffic accuracy and bi-objective problem handling as it was described in sub-section 5.1, student pedestrian traffic model was only established from the university’s optimal route set. however, given that a city displacement patterns depends of external factors, like an specific day, date, near events and diverse social conditions, we consider to model a mobility patterns characterization of the universidad de los andes community, given the multiple circumstances and situations that can affect the displacement patters. also, this research only considered undergraduate students as the only pedestrians that travel the university network, which only considers one group as the university pedestrian traffic. hence, we consider to include displacement patterns that are made from the multiple pedestrian university groups as teachers, staff, etc. as this groups could have a bigger significance on the pedestrian traffic modeling on the university campus. nevertheless, the research objective was to guarantee a student safety based on a covid-19 risk contagion as well a travel displacement optimization in a post-pandemic scenario. therefore, and given the fact that there is no sufficient data to model pedestrian traffic on a pandemic scenario as the date of publishing this work, it is suggested to have in consideration current variables that impact on the pedestrian traffic modeling (like social distancing) with the aim of modeling a displacement pattern as accurate as possible for this new scenarios. 8.3 susceptibility probability function accuracy during the development of the mathematical model and the function that represents the interaction cost (f2), it was noted that the function and the modeling of the susceptibility probability of contracting covid-19 was completely dependent on [5]. as for the readers awareness, it should be noted that the previous reference contains preliminary and ongoing research as the probability function is still being function as the publishing date of this work. it should be noted that, as the date of developing this project, there was lack of studies and modeling data that we could based on for modeling an accurate covid-19 risk contagion probability. therefore, it should be noted that the function calibatrion parameters should be interpreted as placeholder values and the susceptibility probability function relies primary on the encounters students have during their displacements. we consider the importance to validate and implemented accurate and certified probability covid-19 risk contagions functions that rely more on approved researched and have an advance in the development of this topic. paradigmplus (2021) 2:3 34 juan e. cantor et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 future works 9.1 susceptibility probability function parameters calibration based on published future works given the precariousness and early research of the studies published by [5] to define the susceptibility probability of covid-19 as of the date of releasing this paper, it is expected to seek for researches with more exact parameters for the probability function. in future, studies of epidemiological models about the coronavirus may offer more specific functions that model the contagion probability or susceptibility of the virus, which is relevant for the calibration of the interaction costs of the initial problem. 9.2 medical domain approbal for the proposed solution we considered that a validation process must be complemented from the medical domain point of view in the future. in other words, the inclusion and approval of health entities that comprehend the covid-19 contagion processes could help to calibrate and adjust our research to propose a joint solution validated from the computational and medical point of view. 9.3 multi-user model and design this iteration of the project only considered a mono-user scenario interacting with the mobile application. however, the release of a mobile application where multiple users make use of it opens the scenario for including the users application within the calculated pedestrian traffic models. in this case, it is suggested that in addition to including additional pedestrian traffic data in the back end of the mobile application, the optimal routes taken by other users of the mobile application should be included. this allows the incorporation of a pedestrian traffic model that takes into account the multi-user use of the mobile application. 9.4 software architecture impromevement given that the server deployment back end is located on a local machine and network, and based on the consideration of a multi-user scenario of the mobile application, we suggested that the server should be deployed and configured on a cloud computing service platform . it should be considered to establish better response times between server and client, or create communication protocols that do not degrade the user experience when requesting an optimal route or the calculations made by the backend service. authors’ information – juan e. cantor obtained a degree in computer and systems engineer (2020) at the universidad de los andes in bogota, colombia. he currently works as java technical lead at scotiabank colpatria bank. his research interests are mathematical optimization, mobile application development, and software architecture. – germán a. montoya obtained a degree in electronics engineering (2006) at the universidad pontificia bolivariana in medellín, colombia. he also obtained a ph.d. in engineering (2017) at the universidad de los andes in bogota, colombia. currently, he is a postdoctoral assistant at the universidad de los andes in the system and computing engineering department. his main interests are mathematical optimization, simulation, and wireless sensor networks. paradigmplus (2021) 2:3 establishing best pedestrian paths considering sars-cov-2 contagions 35 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . – carlos lozano-garzon received a degree in system engineering (2002) at the universidad nacional de colombia in bogota, colombia. he also received a ph.d. degree at both universidad de los andes, colombia, and universitat de girona, spain in 2017. he is currently assistant professor at the universidad de los andes, colombia in the system and computing engineering department. he worked as a professor at the universidad nacional de colombia, among others, as well as a visiting professor at the universidad nacional de asunción, paraguay. his research interests are network optimization from a multi-criteria perspective, smart mobility: vehicle networks and intelligent transport systems, internet of things, and network security. authors’ contributions – juan e. cantor participated in the conceptualization and methodology as well as the original draft preparation, the mathematical optimization model, the software tests, validations, and final review of the manuscript. – germán a. montoya participated in the conceptualization and methodology as well as the methods review and final review of the manuscript. – carlos lozano-garzon participated in the conceptualization and methodology as well as the methods review. competing interests the authors declare that they have no competing interests. funding no funding was received for this project. references [1] organización mundial de la salud, “q&a. how is covid-19 transmitted,” 2020. https://www. who.int/news-room/q-a-detail/q-a-how-iscovid-19-transmitted. [2] universidad de los andes, “facts and figures.” https://planeacion.uniandes.edu.co/en/ statistics/factsand-figures, 2019. [3] universidad de los andes, “campus en cifras.” https://campusinfo.uniandes.edu.co/es/ campusencifras, 2019. [4] j. e. cantor, c. lozano-garzón, and g. a. montoya, “a multi-objective mathematical optimization model and a mobile application for finding best pedestrian routes considering sars-cov-2 contagions,” in fourth international conference on applied informatics (h. florez and m. f. pollocattaneo, eds.), pp. 191–206, springer international publishing, 2021. [5] s. a. muller, m. balmer, a. neumann, and k. nagel, “mobility traces and spreading of covid-19.” https://www.medrxiv.org/content/10.1101/2020.03.27.20045302v1.full.pdf, 2020. [6] n. kumar, j. b. oke, and b.-h. nahmias-biran, “activity-based contact network scaling and epidemic propagation in metropolitan areas.” https://arxiv.org/abs/2006.06039, 2020. [7] universidad de los andes, “reporte de sostenibilidad 2019.” https://sostenibilidad. uniandes.edu.co/images/reporte2019/reporte-de-sostenibilidad-2019w.pdf, 2019. paradigmplus (2021) 2:3 https://www.who.int/news-room/q-a-detail/q-a-how-iscovid-19-transmitted https://www.who.int/news-room/q-a-detail/q-a-how-iscovid-19-transmitted https://planeacion.uniandes.edu.co/en/statistics/factsand-figures https://planeacion.uniandes.edu.co/en/statistics/factsand-figures https://campusinfo.uniandes.edu.co/es/campusencifras https://campusinfo.uniandes.edu.co/es/campusencifras https://www.medrxiv.org/content/10.1101/2020.03.27.20045302v1.full.pdf https://arxiv.org/abs/2006.06039 https://sostenibilidad.uniandes.edu.co/images/reporte2019/reporte-de-sostenibilidad-2019w.pdf https://sostenibilidad.uniandes.edu.co/images/reporte2019/reporte-de-sostenibilidad-2019w.pdf 36 juan e. cantor et al. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [8] g. j. working group, “geojson.” https://geojson.org/, 2016. [9] m. schultz and j. fuchte, “evaluation of aircraft boarding scenarios considering reduced transmissions risks.” https://www.mdpi.com/2071-1050/12/13/5329, 2020. [10] knime, “knime.” https://www.knime.com/, 2020. [11] firebase inc, “firebase.” https://www.firebase.google.com/, 2012. [12] python software foundation, “python 3.6.” https://www.python.org/, 2020. [13] e. w. dijkstra, “a note on two problems in connexion with graphs,” numerische mathematik, vol. 1, no. 1, pp. 269–271, 1959. [14] y. haimes, l. lasdon, and d. wismer, “on a bicriterion formulation of the problems of integrated system identification and system optimization,” ieee transactions on systems, man, and cybernetics, vol. smc-1, no. 3, pp. 296–297, 1971. [15] w. hart, j.-p. watson, and d. woodruff, “pyomo.” https://pyomo.readthedocs.io/en/ stable/, 2019. [16] world wide web consortium, “web services architecture,” 2004. [17] m. grinberg, flask web development: developing web applications with python. o’reilly media inc., 2018. [18] a. asthana, a. kane, and a. sobti, “postman api client.” https://www.postman.com/product/ api-client/, 2014. [19] google llc, “android pie.” https://www.android.com/versions/pie-9-0/, 2019. [20] jetbrains, “kotlin.” https://github.com/jetbrains/kotlin/releases/latest, 2020. [21] google llc, “maps sdk for android.” https://developers.google.com/maps/ documentation/android-sdk/overview, 2020. [22] freepik company. https://www.flaticon.com/, 2020. [23] google llc, “android volley.” https://developer.android.com/training/volley, 2020. paradigmplus (2021) 2:3 https://geojson.org/ https://www.mdpi.com/2071-1050/12/13/5329 https://www.knime.com/ https://www.firebase.google.com/ https://www.python.org/ https://pyomo.readthedocs.io/en/stable/ https://pyomo.readthedocs.io/en/stable/ https://www.postman.com/product/api-client/ https://www.postman.com/product/api-client/ https://www.android.com/versions/pie-9-0/ https://github.com/jetbrains/kotlin/releases/latest https://developers.google.com/maps/documentation/android-sdk/overview https://developers.google.com/maps/documentation/android-sdk/overview https://developer.android.com/training/volley introduction related work general solution design definition of initial mathematical model interaction cost function pedestrian traffic simulation on university campus pareto front development multi-objective mathematical optimization model interaction cost function pedestrian traffic simulation on the university campus data preprocessing configuration and loading of pedestrians' initial and intermediate nodes pedestrian traffic' displacement pattern pareto front development heuristic bi-objective problem understanding single cost approach to solution heuristic results comparison and validation with the pareto front results comparisons and validation based on the mathematical model mobile application design software architecture validations conclusions performance and limitations university campus graph modeling optimization student pedestrian traffic accuracy and bi-objective problem handling susceptibility probability function accuracy future works susceptibility probability function parameters calibration based on published future works medical domain approbal for the proposed solution multi-user model and design software architecture impromevement vol. 1, no. 1, pp 22 41, 2020 doi: 10.55969/paradigmplus.v1n1a2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . what is game balancing? an examination of concepts alexander becker id 1,� and daniel görlich id 2 1wolfsgrubenweg 25, 67069 ludwigshafen am rhein, germany, alex@aowbecker.de 2srh university heidelberg, ludwig-guttmann-str. 6, 69123 heidelberg, germany, daniel.goerlich@srh.de abstract balancing is commonly considered critical to any game’s success. however, there is no consensus on what “game balancing” actually means. this paper examines fourteen publications by experienced game designers and other renowned authors. the results of a formal semantic analysis of the respective authors’ reflections prove that no two authors share identical understandings of “game balancing.” their differing concepts of the term are analyzed and discussed in order to identify key aspects, similarities, and differences. contrary to what one might expect, concepts such as fairness, flow, or user satisfaction are identified to be important, but no central or fundamental concepts for the definition of game balancing. keywords: game balancing ⋅ video games ⋅ semantics ⋅ concepts received: 13 march 2020 ⋅ accepted: 17 april 2020 ⋅ published: 21 april 2020. 1 introduction good game balancing is considered critical to creating successful games [1]. surprisingly though, there is no consensus on what “game balancing” actually means. in a highly abstract way one can interpret “game balancing” as the activity of tuning a game’s rules, levels, difficulty, numbers, algorithms etc. to achieve desired goals (see, for example, [2]) such as keeping a game winnable, making it fair for all players, keeping it challenging, making it replayable, etc. much has been published about “game balancing”, though rarely from a scientific point of view. considerations on the term and the process of balancing a game most often stem from practitioners. we therefore reviewed fourteen publications by renowned authors and performed a formal concept analysis, comparing these authors’ concepts of “game balancing.” the results prove the authors’ definitions to be diverse, focusing on their respective goals and key aspects of game design. although the authors’ concepts often overlap, especially with respect to central aspects of game balancing, every author has an at least slightly differing view, a different concept, and therefore a unique understanding of game balancing. there are no two authors or teams of authors who agree on completely identical concepts of “game balancing.” this article is distributed under the terms of the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. https://doi.org/10.55969/paradigmplus.v1n1a2 https://orcid.org/0000-0002-1943-2984 https://orcid.org/0000-0002-4275-5466 what is game balancing 23 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 methodology to the best of our knowledge, no extensive semantic or ontological analysis of the term “game balancing” has been published by other authors so far. also, no publications seem to exist addressing game balancing for serious games and other types of games focusing on other goals than fun. few scientific publications such as [1], [3], [4], and [2] focus on game balancing and especially automated or even dynamic game balancing, instead relying on definitions from practitioners or adopting simplified reinterpretations of the term relating to their respective goals (e.g., [2]). in section 3, we therefore present the practitioners’ definitions first before continuing with the state of the art in scientific publications. in order to obtain the fullest possible understanding of the term “game balancing”, we have reviewed the one most important publication of each practitioner or team of practitioners that can be regarded as renowned due to his/her/their influence on the games industry and/or game development education. many of them are experienced game designers and authors of acclaimed standard texts or books, and some of them even teach game design. others are traditional journalists or youtubers focusing on game design, game development or game education. each practitioner or team is briefly introduced in section 3 to prove his/her/their reputation. in section 4 we present and in section 5 we discuss the results of the semantic analysis based on a slightly modified formal concept analysis [5] as described in [6], with additional intermediate steps: while the authors are simply treated as formal objects, their work is condensed into concise formal properties. all properties are connected to their authors and identical properties thus connect to multiple authors. the resulting table can be used to generate lattice diagrams that visualize all connections, and therefore are quantifying consensus and dissensus to a certain degree. an excerpt from this table is provided in table 1. based on these authors’ descriptions, opinions and reflections, more than 500 resulting properties of “game balancing” were grouped into 48 concepts; those 22 concepts that were described by more than one author, were used as the basis for a formal concept analysis. the results were consolidated and visualized using the online lattice editor by [7], employing its “build concept lattice (addintent)” and “min intersection layout (using cenexp algorithm)” settings. the resulting structure was beautified, but otherwise not changed for depiction in fig. 1. a table of the objects, concepts and properties identified in a first formal concept analysis, at that time based on 11 instead of 14 publications, can be found in [8]; an exemplary excerpt from the up-to-date table is printed in table 1. 3 state of the art while ernest adams has worked as a game designer since 1989, he is also a game design consultant and has written several books about the same field. he also is founder of the international game developers’ association (igda). for several years, as in publications [9] and in [10] together with andrew rollings, author of “game architecture and design”, adams has elaborated on the player’s skill, especially in decision-making. balancing should have the goal of keeping skill the determining factor for a game’s outcome despite some chance still being a desirable quality in most games. fairness in player vs. player (pvp) games, having equal chances of winning at the beginning of a match no matter what available options are chosen, and difficulty in player vs. environment (pve) games are substantially connected to this. those pve games should provide constant challenges without ever frustrating the player. keith burgun is an independent game designer, author, composer, visual artist, and author of two books on game design. in [11], he sees the prime objective of game balancing in keeping game elements relevant. this is mainly done to prevent a ruination of the game by dominant strategies and to keep a player’s decision impactful. game elements must be weighed against others within all contexts they compete in. paradigmplus (2020) 1:1 24 alexander becker and daniel görlich . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . david sirlin, president of sirlin games, is a game and graphics designer working mostly on competitive multiplayer games. in [12], he centres his concept of game balancing around providing viable options and fairness in the aforementioned sense to the player. viable options refer to meaningful decisions between promising alternatives. james portnow is a game designer and game design consultant, known for his theories on socially positive game design and as co-founder of the youtube channel “extra credits.” his channel’s episodes present, among others, the concept of perfect imbalance [13]: while a balanced game can easily lead to a stagnation of strategy discovery, since it only becomes a question of executing proven ones, slight differences in power between game elements encourage players to constantly search for new solutions against currently popular strategies. dan felder worked for blizzard entertainment as a game designer on “hearthstone” and later as senior game designer at electronic arts. in [14], he sets the premise that balancing should help games to create a positive experience. this cannot be fulfilled when any game contains broken gameplay, which consists of strategies or even singular actions rendering a lot of decisions meaningless. however, perfect equality would also do that since there would be no reason left to choose any action over another. implementing small power gaps is considered a critical part of game balancing. jeannie novak is lead author and series editor of the “game development essentials” series. similar to rollings & adams in [10], she introduces the reader of [15] to the distinction between static and dynamic game balancing. while all authors emphasize the impact the player’s skill should have on the outcome of a game, those two types of balancing focus on complementary parts of a game: static game balancing is concerned with rules, mechanics, values and their relations to each other, while dynamic game balancing influences how the players’ actions change the course of a running game, and therefore could be considered a form of real-time balancing. richard rouse iii has worked as design and narrative lead for ubisoft and microsoft studios. he establishes in [16] games being a collection of interacting subsystems. changes within one subsystem almost certainly affect other parts of the game as well, so their dependencies must be strongly considered when balancing a game. jesse schell is professor for game design at carnegie mellon university and former igda chairman. in his award-winning book “the art of game design: a book of lenses” [17] he describes multiple contrasting concepts that typically appear in video games, e.g. challenge vs. success, or skill vs. luck. to identify the right middle ground, one has to take the audience into account. to achieve it, one can manipulate not only game mechanics but their combinations and relations as well. ian schreiber began programming and designing games in 2000, has written two books on games, is co-founder of the global game jam and assistant professor for interactive games and media at rochester institute of technology. in [18], schreiber uses a primarily mathematical approach towards game balancing. he calls everything within a game quantifiable to some degree. however, numbers only have a meaning within their given context(s) and have no meaning if not related to anything. game balancing is not only concerned with this, however, but also with the players themselves: games offer different levels of information access, players have differences regarding their ability to process information and expectations. even factors beyond the actual game can influence their behavior. tynan sylvester, founder of ludeon studios, has been designing games since 2000 and has written an instructional game design book [19]. to achieve the prime goal of creating and keeping multiple viable strategies, he states that one must work on the relative differences in power game that elements have. strategies are considered viable if all of them have similar chances of leading to a victory. marc brown is a game journalist and creator of the “game maker’s toolkit” youtube channel and series. in [20], he focuses on fairness in multiplayer games and how those must not only provide a balance of game elements, but of all options a game offers, including singular actions and strategies. additionally, the players’ perception of balance is just as important as the actual balance. balancing does not stop there though; one also must measure and interpret the state of balance correctly and should communicate changes accordingly to players to prevent misconceptions. paradigmplus (2020) 1:1 what is game balancing 25 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . rym decoster and scott rubin host the “geeknights” talk radio show. on the pax south 2018 [21], they presented an approach for balancing multiplayer games around the player instead of mainly mechanics and numbers. among other things, this includes how to match players accordingly. adam millard runs a youtube channel named “the architect of games”, which started in 2011. in [22], he does not exactly try to find a new definition of game balancing, but instead shifts the focus from providing a perfectly balanced game to preserving an equilibrium of power within a game. according to him, a developer’s obligation is to keep exactly that equilibrium intact and not to achieve numerical perfection. there are only a few scientific publications on game balancing. they use or extend the term, often adopting definitions from practitioners: andrade et al. [1], for example, references raph koster’s “a theory of fun for game design” [23], concluding that “game balancing aims at providing a good level of challenge for the user” and keeping the player interested in playing the game. they evaluate dynamic game balancing approaches, stating that those must satisfy at least three basic requirements: adapting to the player’s initial level, tracking the evolution and regressions of the player’s performance, and keeping the game “believable.” their evaluation indicates that adaptive approaches may be more effective, resulting in better user satisfaction than traditional, non-adaptive, pre-defined game balancing using static difficulty levels. similar to andrade [1], tijs et al. [3] state that dynamic game balancing requires adjusting a game’s parameters such as the difficulty level to a player’s skill level while playing: to be enjoyable, these parameters should be changed over time to avoid undesired player emotions, such as boredom and frustration. they argue that the player’s emotional or affective state must be considered by dynamic game balancing, so that the game can become emotionally adaptive. lankveld et al. [4] also define game balancing as the adaptation of the game difficulty to a player’s skill or, more precisely, the relationship between a game’s complexity and a player’s abilities: the player should be challenged, but not frustrated by the complexity of the game. they suggest a different approach: they apply the incongruity theory, trying to avoid letting the difference between the complexity of a game and the complexity assumed by the player’s internal human model becoming too large. in order to demonstrate the feasibility of automating the game balancing process, volz et al. [2] treat game balancing as a multi-objective optimization problem, assuming that “game balancing is an important part of the (computer)game design process, in which designers adapt a game prototype so that the resulting gameplay is as entertaining as possible.” similar to schreiber [18], they define game balancing “as the modification of parameters of the constitutive and operational rules of a game (...) in order to achieve optimal configurations in terms of a set of goals, i.e. a parameter tuning problem.” 4 semantic analysis for conceptual structuring although the previously introduced authors may not seem to be far apart, their ideas of game balancing differ in many details. most obvious are their different perspectives and scopes: some focus on game design, others already consider more technical perspectives or even reduce game balancing to adjusting or optimizing numbers. many authors use terms such as “fairness” or “difficulty” that may also be difficult to grasp. while all authors emphasize different aspects of game balancing, they rarely contradict each other. there are some concepts they commonly agree on, despite sometimes using divergent names or descriptions. examples are dominant vs. degenerative strategies and (meaningful) decisions vs. (viable) options. to analyze and compare the presented concepts of game balancing, we therefore need to understand the authors’ basic ideas, concepts and assumptions that some of them take for granted, but others try to define or describe explicitly in their respective publications. adams [9, p. 403] links balance to the player’s skill: “in the most general sense, a balanced game is fair to the player (or players), is neither too easy nor too hard, and makes the skill of the player the most important paradigmplus (2020) 1:1 26 alexander becker and daniel görlich . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . factor in determining her success.” adams adopts fairness, an inherently subjective concept, as a general goal, and links fairness to the player’s “skill”—another term that requires more consideration. to adams, it is generally important that chance must not be powerful enough to make skill irrelevant. to achieve that, game design should allow the player to make meaningful decisions, so that the outcome of the game primarily depends on the player’s choices [9, p. 404]. game design must therefore avoid dominant strategies, i.e. strategies that do not need to be infallible but are so strong that they leave players no reason to use any other strategy. even singular decisions or the mere avoidance of losing the game can be considered dominant strategies [9, p. 405], as well as so-called “exploits” which trivialize parts of a game [9, p. 410f]. dominant strategies make alternatives worthless and render the player’s decisions meaningless [9, p. 405f]. sylvester calls dominant strategies “degenerative strategies” [19, p. 160] and warns that adding new game elements does not always create more meaningful decisions; indeed, it might do just the opposite [19, p. 159]. depending on the (in-)transitivity of involved game elements, adams sees numerous ways to avoid dominant strategies. “transitivity” applied to relations between game elements means there is a transitive order in power or usefulness: if game element a outclasses b and b outclasses c, a also outclasses c. in the opposite case of intransitivity, every game element can be beaten by some other, which supports the avoidance of dominant strategies. to avoid dominant strategies within a transitive relation, one can change costs or use positive feedback. costs might be outright stated prices, but also “shadow costs”, such as hidden detrimental properties [9, pp. 406–407]. positive feedback describes giving the player rewards to accelerate future progress [9, p. 408] without creating a sure-fire course [9, pp. 429–430]. furthermore, adams [9] recommends assessing intransitivity in the elements’ properties, instead of defining superiority, as is the case in rock-paper-scissors. this can be achieved by giving elements unique traits that cannot be solely compared by their values. adams calls this “orthogonal unit difference” [9, pp. 408–410]. besides general criteria, adams distinguishes between priorities for pvp and pve games: those are fairness and remaining able to win throughout the match for pvp [9, p. 404] vs. difficulty for pve games [9, p. 405]. while playing a match in a pvp game, any player should have opportunities for a comeback, if he or she falls behind early on. stalemates, the inability to win or finish the game properly, should be avoided [9, p. 404]. to adams, fairness is of paramount importance to pvp titles. he interprets fairness as all players having a roughly equal chance of winning at the start of every game. fairness should always be granted [9, p. 404], e.g., by accordingly balancing starting options, by protecting newer players, or by providing game updates [9, pp. 414–416]. in symmetrical games that provide identical starting conditions to all players, fairness is already given. most games, however, come with different options at the start of a game and are called asymmetrical, which makes them more prone to developing dominant strategies [9, p. 413]. sirlin also adds a scale to the concept of (a)symmetrical games: while symmetry is a fixed condition, a game can be more or less asymmetrical. the more asymmetrical a game is, the more important maintaining fairness becomes [12, p. 1]. difficulty in pve games is strongly connected to the players’ expectations and requires consistency [9, p. 405]: sudden jumps should be avoided as well as stalemates and losses the player could never have prevented. also, important decisions should be marked as such and all necessary information to finish the game should be accessible [9, pp. 416–417]. all this serves the purpose of bringing the player into a state of flow. however, expectations of and requirements for this can vary immensely between audiences [9, pp. 418–419]. adams subclassifies “difficulty” into: 1. absolute difficulty. the combination of required skill and time restrictions. it highly correlates with the game’s numbers, such as enemies’ strengths and maximum health points [9, p. 420]. 2. relative difficulty. what is left when the player’s power is subtracted from the absolute difficulty [9, pp. 420–421]. 3. perceived difficulty. what the player experiences. also takes his or her expertise into account [9, p. 422]. paradigmplus (2020) 1:1 what is game balancing 27 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ideally, the perceived difficulty stagnates or increases throughout the game according to the target group’s preference. the relative difficulty must grow faster, since the player’s experience also increases. the absolute difficulty must rise even faster, if the player gains more power [9, p. 42324]. also, if the player can adjust the difficulty somehow, the game covers a wider audience [9, pp. 425–426]. while burgun [11, p. 2] primarily addresses multiplayer games, he includes single player games in most of his statements, especially if they are supposed to have replay value. generally, a wellbalanced game stays interesting for longer. his definition focuses on choices [11, p. 1]: “gameplay is all about making choices and in a poorly-balanced game, many of the choices available to the player are essentially rendered useless.” he argues, as does adams, that dominant strategies can render other strategies useless and game elements irrelevant. whether a strategy is dominant or not, however, can depend on the player’s skill, especially in more complex asymmetrical games. therefore, the target group should be considered when setting a balancing goal. while tailoring to all audiences would be ideal, this is too intricate. the effort rather should focus on one group, e.g., either competitive or casual players [11, p. 2]. furthermore, game elements must be balanced within all contexts they operate in. this includes, for example, costs compared to other elements in the same production line, their usefulness when competing with opposing elements, but also their power in relation to others that prohibit or limit each other’s use [11, p. 1]. this contest on multiple layers takes place even in symmetrical games, although they typically include fewer contexts, since all players use the same selection of elements [11, p. 2]. james portnow of extra credits summarizes his concept of perfect imbalance as follows [13, 0m34s]: “fundamentally, it is the idea in game design that you don’t always want things to be perfectly balanced. in fact, in most games you actually wanna make sure that there are some imbalances in your systems. (...) many games are actually made far more engaging by just a little bit of imbalance, multiplayer games especially.” portnow highlights that those imbalances refer to subtle differences, not huge gaps in power [13, 0m30s–1m00s]. the distinction between “out of balance” and “broken” is crucial: no strategy or game element should be much stronger than the rest, but only slightly. these subtle variations in power can be measured and adjusted with the help of a “power curve”, a relation from cost to power for game elements [13, 3m35s–4m20s]. in the metagame, which is basically the situation or context in which the game is played, such an imbalance allows for discussions, discovery and therefore fun, while players try to find slightly advantageous strategies. they can even discover new solutions for problems without having to know all established strategies. the resulting metagame allows players to grow into the game and prevents any playstyle from becoming the best [13, 2m45s–3m35s]. in contrast to a perfect imbalance, in (almost) perfectly balanced games like chess, the best strategies are discovered sooner or later. at that point, most players are left with nothing but executing those instead of creating their own strategies. only the best players could still find new strategies that are not already established. this removes part of the fun for a huge portion of the audience [13, 1m00s–2m45s]. felder [14] bases his view of balancing on the avoidance of imbalance: “i’ve found that the most useful definitions of balance are based on what we’re trying to avoid: broken gameplay. (...) when you’re designing a game you naturally want to create a positive experience for your players. when your gameplay isn’t providing that experience, your game is broken. it’s that simple.” an example for something broken would be a strategy that makes all others obsolete, and therefore a lot of decisions, if not all of them, as well. however, the exact opposite would also render decisions useless: if every option was equally good, there would be no reason to prefer one. instead, subtle differences in power allow for strategic decision making. novak [15, p. 202] relates balancing to the player’s skill: “the ease of winning the game also increases as the players’ skills increase. random events (...) can still occur in the game that might decrease a skilled player’s chance of winning. however, a better player should be more successful in general at the game than a less-skilled-player, unless the game is based purely on luck instead of skill.” considering the desired results paradigmplus (2020) 1:1 28 alexander becker and daniel görlich . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . of balancing, novak concludes: “a game is balanced if players perceive that it is consistent, fair, and fun,” implying that the perceived quality of game balance may be subjective. without using potentially subjective terms, rollings and adams [10, p. 240] similarly state that “a balanced game is one where the main determining factor for the success of the player is the skill level of that player. that does not mean that random events cannot occur, but a better player should ordinarily be more successful than a poor one unless he has an unusually long run of bad luck.” while not identical, novak [15] and and rollings and adams [10] have a similar focus. they name criteria found in well-balanced games: – the game becomes consistently more challenging [15, p. 202ff] until the climax [10, p. 272]. – the player perceives fairness by always being able to win, even after early mistakes [15, p. 202ff]. this perception extends to offering the player information, control, training and beginner protection. unnecessary repetition of tasks should be avoided [10, pp. 272–276]. – no trivial decisions. these are decisions that have no impact, or one alternative is clearly the best [15, p. 202ff]. additionally, micromanagement – the act of administering a high number of small elements – should never be mandatory, only optional [10, p. 276f]. – no trivial decisions. these are decisions that have no impact, or one alternative is clearly the best [15, p. 202ff]. additionally, micromanagement–the act of administering a high number of small elements–should never be mandatory, only optional [10, pp. 277–279]. – adjustable difficulty [15, p. 202ff] [10, pp. 279–281]. beyond general criteria, they continue their investigations of static and dynamic balancing. static balancing is concerned with the game’s rules and how they interact. its main goals are to avoid dominant strategies [10, p. 243] and provide fairness [10, p. 267]. rollings and adams [10] contribute two new aspects to dominant strategies: a strongly dominant strategy always wins the game, while a weakly dominant strategy prevents a loss. other than the former, the latter can be beneficial to the game. furthermore, an “almost dominant strategy” does not always win the game but is still the best option available under any circumstances [10, p. 244ff]. novak [15] describes another strategyrelated concept a game should include, so-called “obvious strategies”. these are strategies that are generally the best, but explicitly not under any circumstances [15, p. 203]. game elements with transitive relationships to each other are useful as rewards for progress [15, p. 204] [10, p. 252ff], while those with intransitive relationships profit from having orthogonal unit differences [10, pp. 228–261]. dynamic balance covers how the game changes over time and with player interaction [15, p. 207]. it can be done passively or actively to prevent unfair advantages [10, p. 267]. players interact with the dynamic balance in three ways [15, p. 207]: 1. restore balance. the player restores a disturbed balance. de-balancing forces are weaker than the player or can be removed [10, p. 269]. 2. preserve balance. the player repels de-balancing forces. those are at least equally as strong as the player. there is no win condition, and the player loses when doing nothing [10, p. 269ff]. 3. destroy balance. the player alternates the state of balance or creates chaos [10, p. 270ff]. rollings and adams continue with “emergence”, the creation of complex results using simple rules. a world can be simulated via a reduction to basic properties and still deliver the targeted experience. this reduces complexity and preserves control for the player but might favor the formation of dominant strategies [10, pp. 262–265]. there is also a reverse version of positive feedback called negative feedback, which limits progress or leadership, e.g. by adding further costs [15, p. 206] [10, pp. 265–267]. paradigmplus (2020) 1:1 what is game balancing 29 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . rouse [16, p. 493] describes games as systems composed of subsystems where “all of these [sub]systems must be in place, since changing one system impacts how the other systems must be setup in order to achieve the overall balance you are seeking.“ he distinguishes between provisional balancing creating a balanced foundation, and true balancing using this basis mainly to adjust numbers to deliver a quality game. play tester feedback is crucial for this iterative approach [16, pp. 493–494]. it is also necessary to understand the relations and influences within and between one’s own systems to avoid unintended consequences, while being able to quickly iterate values within the game [16, p. 495]. schell [17, p. 202] states that “balancing a game (...) is all about understanding subtle nuances in the relationships between the elements of your game and knowing which ones to alter, how much to alter them, and which ones to leave alone.” he continues with various patterns that frequently appear and are partially opposed to each other. some of them are: – challenge vs. success. a game should neither be too easy nor too difficult. the goal is to keep the player in a state of continuous flow. however, learning rules and controls are already challenges in themselves [17, pp. 207–209]. – meaningful decisions are an integral part of a gaming experience. however, any decision becomes meaningless if one of its options is clearly the best or the decision has no impact at all [17, pp. 210–214]. – skill vs. chance. it is commonly good to have both in a game to a certain degree, dependent upon the audience [17, p. 214f]. – strategy vs. dexterity. the best ratio depends on the audience [17, p. 215ff]. – competition vs. cooperation. although many games contain just one, they can appear combined [17, pp. 216–218]. – short vs. long. via changing win conditions, players should have enough time to strategize, but never get bored [17, p. 219]. – rewards vs. punishment. rewards are a human desire. the rewards must increase over the course of a game to keep the player interested [17, pp. 219–222]. although rewards should generally be preferred, punishment can also enhance the experience, e.g. by increasing the challenge [17, pp. 222–225]. – simple vs. complex. it is generally better to create complex results using simple rules, but small case-specific additions might still be beneficial. the former makes a game “elegant” and can be referred to as “natural balancing”, while the latter is called “artificial balancing”. games should neither be unnecessarily complex nor simple to the point of being trivial [17, pp. 226–230]. despite all details being important, schell emphasizes the big picture: the game should feel right. if it does not, one should ask and search for the causes [17, p. 237]. schreiber follows a mathematical approach: “while perhaps an oversimplification, we can say that game balance is mostly figuring out what numbers to use in a game” [18, p. 1]. it is important to understand that every game contains numbers, even if these are not stated outright [18, p. 1], and that numbers only have a meaning within a context [18, p. 2]. these numbers are connected within greater systems that are divided into subsystems, like combat or economy [18, p. 2]. he differentiates between “deterministic” and “non-deterministic” games. the former always produce the same outcome dependent on the action in a certain state; the latter do not, due to chance or other players’ actions. “solvability”, which means that in every situation, there is a recognizable best action, would generally be an undesirable to have in a game, since it renders decision-making obsolete, at least when a player is capable of doing so in real-time. there are different levels of solvability, starting with trivial, which can be solved in real time. others are theoretically solvable but require paradigmplus (2020) 1:1 30 alexander becker and daniel görlich . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . too much calculation to be solved within an acceptable time frame. however, even non-deterministic games are solvable, because even random events have defined outcomes and an expected value. while every game that provides perfect information about its state is theoretically solvable, one can, for example, limit the players’ access to information. this might be affected by the metagame [18, p. 1]. another dimension to balancing is costs. costs are everything that limits access to advantages or interferes with them. advantage is essentially everything that benefits the player. we can view costs as negative advantages and vice versa. they can be calculated as such to create a cost curve [18, p. 3]. schreiber adds that shadow costs can be divided into two concepts: sunken costs and opportunity costs. the former describes prerequisite costs that are indirectly related to the stated costs, while the latter limits the player’s future possibilities once s/he has spent the stated costs [18, p. 6]. schreiber also adds to the concept of rewards. regularly giving out smaller rewards should be preferred to fewer larger ones. players prefer randomization, as long as it still feels like the result of their actions. rewards and progress should support each other, and popular rewards should not be held back [18, p. 7]. sirlin [12, p. 1] focuses on balancing multiplayer games: “a multiplayer game is balanced if a reasonably large number of options available to the player are viable—especially, but not limited to, during highlevel play by expert players.” in addition to fairness, “viable options” must be provided. this refers to meaningful decisions between promising options and excludes the existence of dominant strategies. meaningless decisions unnecessarily increase complexity; only meaningful ones benefit a game [12, p. 1]. however, despite the need for fair starting conditions, a game does not have to be completely balanced within every state of the game. on the contrary, imbalanced states can be situationally desired, e.g. by giving rewards for playing well. he continues that “checkmate situations”, in which one player is clearly going to lose but the game technically does not end, should generally not arise. although manifold games are generally desired and provide many benefits such as counter elements and actions, no game element should be worthless. worthless elements never contribute to making decisions meaningful [12, p. 2]. sirlin also discusses the concept of fairness: comeback mechanics are helpful to prevent unforeseen balancing issues. also, new game elements should be introduced after a balanced basis is created, and banning certain elements or combinations subsequently, e.g. for tournaments, is highly undesirable. when a power curve is used, it should be noted that the exact ratios between power values are not necessarily provided. also, decisions, that are made at the beginning of a match or game should not determine the outcome right away [12, p. 3]. sylvester [19, p. 155] extends the importance of power from game elements to strategies: balancing “means adjusting game mechanics to change the relative power of different tools, units, strategies, teams, or characters.” in addition to being fair, games need to be deep which refers to meaningful play even at higher skill levels: even experts must be uncertain as to the best option in any given situation. to achieve this, strategies should be balanced. a strategy is a specific combination of actions a player can choose from to achieve a certain goal [19, pp. 157–158]. multiple viable strategies must be offered; even two can be enough though, since adding more decisions, even meaningful ones, can increase complexity without making the game deeper [19, pp. 161–162]. a difficulty in balancing strategies comes from having to change game elements which are typically part of multiple strategies, in order to affect a strategy [19, p. 166]. marc brown [20, 0m15s–0m28s] focuses on fairness in multiplayer games: “balance is the art of making sure that all options in a multiplayer game are fair. so, none are underpowered and thus pointless to use. and none are overpowered, and thus dominate everything else.” this mostly concerns actions, strategies and elements within asymmetrical games that feature a wide variety of different options that must be viable among players of roughly the same skill level [20, 0m20s–1m3s]. however, the players’ perceptions of balance are at least as important as the numbers themselves [20, 1m18s–1m27s]; since any changes affect all players, they should be communicated to them to prevent misconceptions [20, 12m25s–12m52s]. to achieve the desired balance, developers can use a variety of means. brown starts with what paradigmplus (2020) 1:1 what is game balancing 31 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . he calls trade-offs, which basically describe the idea that everything advantageous must also come with equally large drawbacks to fit the same power budget [20, 1m46s–2m24s]. while mathematical balance is important, variety is as well, although it makes game generally harder to balance [20, 3m6s–3m36s]. another way to balance games are counters to everything, ranging from actions over game elements to strategies. those create counter-play by creating a cyclical structure similar to rockpaper-scissors. this helps prevent all elements, actions and strategies from becoming overpowered or useless and encourages players to use different actions, elements and strategies, even if a few of them are somewhat unbalanced [20, 3m37s–7m13s]. to balance a game even further, different means such as matchmaking (pairing similarly skilled players), negative feedback loops, catchup mechanics, deor increasing the influence of luck, and alternated playstyles may cautiously be introduced [20, 12m57s–14m18s]. another issue brown [20] adds is the verification that a game is actually balanced. this might require feedback and statistics. however, a global win rate alone is not an indicator since that can be influenced by various factors like usage, skill levels, reliability and some game elements or strategies simply being more or less fun [20, 7m23s–9m44s]. developers might also look into the player-created meta, which essentially is what players consider to be effective at any given time due to sharing information outside of the actual game or its rules. the meta can also help in balancing a game since it encourages players to find counters against popular things [20, 9m45s–10m47s], not unlike the cyclical imbalance described by portnow [13]. but even with a meta a game requires balancing from the developers. although it can be easy to find out when something is dominating or useless by looking at numbers, it generally is more difficult to discover why that is. but as soon as the source is found the developers might “nerf” or “buff” something: a nerf can be achieved by reducing the power of something or increasing the power of its counters, and buffing can be done by increasing something’s power or reducing the power of its counters. buffs should generally be preferred though [20, 10m48s–12m1s]. rym decoster and scott rubin [21, 42m44s–44m40s] propose rating players instead of or in addition to balancing the game: “a balanced game is one in which all players are rated and ratings between two players predict the outcome.” they make clear early on that fun, which is important for players and spectators alike, typically is the opposite of balance and fairness [21, 3m41s–5m31s]. the authors state that asymmetric and somewhat imbalanced games would actually be more exciting for most people [21, 11m41s–12m49s]. also, inherently imbalanced games can be balanced by changing the player order, win conditions or tournament rules, and not only in the level of mechanics and numbers or using power measurements, although all options in a game still should be viable in some way [21, 12m50s–22m14s]. the power of any game element must also take the difficulty of using or playing them into account, e.g. something easy to play should not be stronger than something that is difficult to play. while power and difficulty of using that game element can be correlated, a game cannot be well-balanced and still contain a huge margin of difficulty regarding the use of game elements or strategies [21, 26m12s–30m50s]. however, the focus of balancing should not lie on quantifiable perfection but rather on balancing the players, so the better player always wins, at least when there are no means in place to align players. some form of matching players according to their skills is crucial so that all players consistently win around fifty percent of their games [21, 31m38s–39m4s]. this also has the problem of players almost not noticing that they are getting better since the challenge increases accordingly [21, 39m5s–42m44s]. alongside matchmaking, this can also be achieved by using handicaps (preferably positive ones) to create a somewhat competitive experience between players of different levels of skill, despite that being more suitable for casual games [21, 44m41s–48m09s]. in any case, this balancing of players also has to take factors that lie more or less outside of the game into account, like the players’ access to money input, gaming machines, geographical location, etc., so players do not get totally separated from each other [21, 48m11s–56m28s]. also, developers, especially of competitive games, should be open about balancing discussions to their players so they can expect solutions to existing problems [21, 58m17s–59m45s]. paradigmplus (2020) 1:1 32 alexander becker and daniel görlich . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . while adam millard [22, 11m19s–11m32s] does not exactly provide a definition of game balancing, he relocates its focus: “why is balancing games so difficult? (...) because game balance works in equilibrium, it’s impossible to change just one thing. instead, any balance adjustment creates a ripple effect that changes the whole game.” although he does not state something that was not already conceptualized by one of the previous authors, he, while also partially referring to sirlin, emphasizes the equilibrium of counters of each other that exists within every game, similar to portnow and brown in many regards. it is not the obligation of the developers to create a perfectly balanced game, but only to make sure everything within the game can be countered by at least something. to keep the equilibrium intact, changes must be carefully implemented. 5 discussion having analyzed the semantics behind the above mentioned authors’ concepts of “game balancing”, we will now look for key aspects, similarities and differences in their concepts and will then move on to discuss them. a first version of this formal concept analysis of, at that time, eleven publications, including all tables of concepts, properties, and formal objects, is available in [8, pp. 163–202]. the up-to-date version comprises a table with around 600 entries. table 1 is just a short, but exemplary excerpt. table 1: exemplary excerpt from the around 600 entry-long table of formal properties author’s formal formal objects: game balancing according to concepts properties ad no ra ro sll sy bu po fe sr si b dr m balancing repeatedly iterating and adjusting is the best method. ra ro during balancing, numbers of the game are adjusted. sy sr issues should be approached where they arise. sr si characteristics of a wellbalanced game early mistakes do not result in a loss later and do not prevent victory in later stages. no ra chance is not powerful enough to render skill irrelevant. ad no ra multiple strategies are viable even if they have smaller differences in power. fe difficulty the player should be able to adjust difficulty in some way. ad no ra sr goal of difficulty. achieving and maintaining a constant flow state. sll sr paradigmplus (2020) 1:1 what is game balancing 33 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . the optimal difficulty depends on the audience. sll sr symmetry symmetric games offer players equal starting conditions. ad no ra sll sy bu sr si b dr symmetric games are automatically fair. ad sll sy sr dr in symmetry, only player skill matters. ra game elements still must be balanced. bu sr the results are further visualized in figure 1 and described in table 2. figure 1 shows which author covers which concepts, and which concept is covered by which author. also, coupled occurrences can be identified. the higher up a node appears in the diagram, the more authors and fewer concepts it contains, and vice versa. only the green nodes are labeled, since they mark the fundamental level on which an author or concept occurs. whatever term is in bold in table 2 marks the first appearance of that concept or author. for an author this means that this node contains all concepts covered by this author and every directly or indirectly connected node above contains that author as figure 1: conceptualization of “game balancing” as a lattice form diagram paradigmplus (2020) 1:1 34 alexander becker and daniel görlich . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . well. for a concept this node shows all authors who cover it. additionally, all nodes connected to this one from a lower level contain that concept as well. smaller nodes indicate a combined appearance and can be used to follow connecting lines. table 2: descriptions for fig. 1. nodes are sorted by the level they appear on, from top to bottom level nodes author’s concept(s) 1 6 adams, decoster & rubin, novak, rollings & adams, rouse, schell, schreiber difficulty 17 brown, burgun, decoster & rubin, felder, millard, portnow, rollings & adams, rouse, schell, schreiber, sirlin, sylvester balancing 19 adams, decoster & rubin, brown, burgun, novak, rollings & adams, schell, schreiber, sirlin, sylvester symmetry 31 adams, brown, burgun, decoster & rubin, felder, millard, novak, rollings & adams, sirlin characteristics of well-balanced games 2 16 adams, brown, novak, rollings & adams, schell, sirlin, sylvester fairness, symmetry 21 brown, burgun, decoster & rubin, novak, schell, sirlin difficulty 32 brown, burgun, decoster & rubin, felder, millard, rollings & adams, sirlin balancing, characteristics of well-balanced games 33 adams, decoster & rubin, novak, rollings & adams, schell, schreiber chance, difficulty, symmetry 3 18 brown, rollings & adams, schell, sirlin, sylvester balancing, fairness, symmetry 23 adams, burgun, rollings & adams, schell dominant strategies, symmetry 34 adams, novak, rollings & adams, schreiber chance, difficulty, intransitivity, positive feedback, symmetry, transitivity 44 novak, rollings & adams, schell, sirlin fairness, symmetry, trivial / meaningless decisions 4 30 brown, portnow, schreiber balancing metagame 35 novak, rollings & adams, schreiber chance, difficulty, intransitivity, negative feedback, positive feedback, symmetry, transitivity 36 novak, schell, schreiber chance, difficulty, economies, symmetry paradigmplus (2020) 1:1 what is game balancing 35 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 adams, rollings & adams, schreiber chance, costs, difficulty, intransitivity, positive feedback, symmetry, transitivity 51 burgun, schell, sirlin asymmetry, balancing, meaningful decisions, symmetry 5 1 adams, rollings & adams chance, characteristics of well-balanced games, costs, difficulty, dominant strategies, fairness, intransitivity, positive feedback, symmetry, transitivity 4 novak, rollings & adams chance, characteristics of well-balanced games, difficulty, dominant strategies, dynamic balance, fairness, intransitivity, negative feedback, positive feedback, static balance, symmetry, transitivity, trivial / meaningless decisions 8 rouse, schreiber balancing, difficulty, games as systems 41 schell, schreiber balancing, chance, difficulty, economies, rewards, symmetry 54 schreiber, sirlin balancing, solvability, symmetry 6 3 novak asymmetry, chance, characteristics of well-balanced games, difficulty, dynamic balance, economies, fairness, intransitivity, negative feedback, positive feedback, static balance, symmetry, transitivity, trivial / meaningless decisions 5 rollings & adams balancing, chance, characteristics of well-balanced games, costs, difficulty, dominant strategies, dynamic balance, fairness, intransitivity, negative feedback, positive feedback, static balance, symmetry, transitivity, trivial / meaningless decisions 14 schell asymmetry, balancing, chance, difficulty, dominant strategies, economies, fairness, meaningful decisions, rewards, symmetry, trivial / meaningless decisions 29 burgun asymmetry, balancing, characteristics of well-balanced games, dominant strategies, meaningful decisions, symmetry 42 schreiber balancing, chance, costs, difficulty, economies, games as systems, intransitivity, metagame, negative feedback, positive feedback, rewards, solvability, symmetry, transitivity paradigmplus (2020) 1:1 36 alexander becker and daniel görlich . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 sirlin asymmetry, balancing, characteristics of well-balanced games, fairness, meaningful decisions, solvability, symmetry, trivial / meaningless decisions 64 brown asymmetry, balancing, characteristics of well-balanced games, fairness, metagame, symmetry 70 decoster & rubin asymmetry, balancing, chance, characteristics of well-balanced games, difficulty, symmetry there are a few important points to note when using the diagram, the table and data contained therein to draw conclusions: while it can be used to draw positive conclusions about connections and semantics, the reverse cannot be accomplished: just because an author is not connected to a concept does not mean that the author never mentions this topic or never uses the according term. it only shows that if they make mention of it, this happens within the context of another concept. it is usually only touched briefly explained there. the diagram reveals countless implications, among them: no concept is covered by every author, which means that there is not even a consensus about one concept being the singular essential core of the term “game balancing.” no author covers all concepts; instead, schreiber, novak, schell, rollings and adams, sirlin, decoster and rubin, brown and burgun each introduce combinations of concepts no other author covers. these implications corroborate our original impression that there is no comprehensive definition of the term “game balancing.” there is not even a common foundation or central core concept for the term, although often several authors employ the same or similar concepts. surprisingly, concepts such as fairness, flow, or user satisfaction appeared to be important, but there were no central or fundamental concepts for the definition of game balancing. however, there is a basis of only four concepts that are not exclusively covered together with specific other concepts, but which are shared by many authors and connected with many other concepts: difficulty, balancing (as a purposive act or process), symmetry, and characteristics of well-balanced games. therefore, we will now continue discussing these qualitatively while also referring to semantically related or stated concepts. characteristics of well-balanced games: while various characteristics meet some explicit approval, such as avoiding stagnation (3/9), allowing the user to make meaningful decisions (3/9) and chance not being powerful enough to render skill useless (3/9), other aspects are still vague. one such aspect is when exactly something becomes “stagnant”: even though the player should not have to unnecessarily repeat tasks, no boundaries are given as to when repetition becomes a problem; after all, games do feature tasks that can or must be repeated in order to progress or improve something. while examples are given such as having to repeat easy parts, there is no rule or definition that applies generally. additionally, there is no comprehensive definition of the commonly used term “skill”: though there is an implicit distinction between skill as “decision-making” and skill as “dexterity” in executing actions, skill is rather the presumed entirety of influence a player has on the course of a game rather than a well-defined term, although it is also noted that the player should have the experience or feeling of being responsible for the outcome of his or her actions. this leads to further problems within the concept of “difficulty.” besides that some authors (3/9) question how developers should measure and interpret the current state of balance: while it is partially agreed that every player of a multiplayer game having a win ratio of around fifty percent is a good sign, this alone cannot measure whether the game is actually balanced. most authors name different additional criteria that should be fulfilled, such as the ones already mentioned, but also the importance of counter options, of a functioning matchmaking system or of providing viable options. beyond that the authors more or less implicitly or indirectly agree that huge imbalances are undesirable, but which level of imbalance is acceptable or paradigmplus (2020) 1:1 what is game balancing 37 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . even beneficial seems to be a controversy. examples for this include which decisions and how many of them should be meaningful and whether providing counter options to actions, game elements or strategies can be enough to compensate at least some balancing issues. concerning meaningful decisions, all authors (3/3) underline their importance and describe meaningless decision-making as useless or even harmful to a game. however, there is no exhaustive definition of what a “meaningful” decision exactly is; instead, it is circularly defined as not being meaningless. the type of meaningless decisions that only provide weak options plus one option that is clearly the best, can also lead to dominant strategies. if these are obviously dominant, sylvester [19] even calls them “degenerative strategies.” those authors who talk about any sorts of strategies agree that dominant strategies should usually be avoided (7/7), with only few exceptions. difficulty: which difficulty level may be right is assumed to depend on the players’ perception, the players’ skills and how quickly their skills improve (5/6). although skill and the individual perception of skill are somewhat subjective quantities, they must be mapped onto in-game numbers that can be calculated and manipulated. several authors (2/6) explicitly name the goal of keeping the player in a state of flow. in order to achieve this, the perceived difficulty must be right, by staying the same or steadily growing. this therefore is somewhat dependent on the audience and even extends to aspects that are not usually considered part of balancing, like learning controls, accessibility and presentation. additionally, decoster and rubin [21] are the only ones to emphasize a form of difficulty important for multiplayer games, while all other authors focus on difficulty in single player games: power can be connected to or correlated with the difficulty of executing it. an extremely easy-to-use game element or strategy should not be powerful as well. this also would lead to any multiplayer game not being able to be balanced and also contain a huge variance of difficulty. symmetry: many authors (6/10) explicitly state that symmetrical games are automatically fair. many modern games, however, are typically asymmetrical, which leads to a higher balancing effort. sirlin [12] appears to be the only author to explicitly state that a game can be more or less asymmetrical. fairness is an important concept, most authors agree. however, adams [9] and sirlin especially emphasize the importance of fairness for multiplayer games, and also state that fairness alone does not provide a well-balanced game. additionally, symmetrical games are considered to becoming boring quicker than asymmetrical ones and while they are easier to balance, their game elements still must be balanced. balancing: the high number of methods and means for balancing games may be an indicator for the lack of agreement on the actual goal of balancing, i.e. the pursued state of game balance: while no author denies that huge imbalances are bad, smaller ones can be accepted. not only that: portnow [13], decoster and rubin [21], millard [22] and felder [14] even state that a slight imbalance might be more beneficial than perfect balance. they primarily base their view on a perfectly balanced game offering little to no reason to try something other than proven strategies, or all options being equally good; therefore, the decision between those options does not matter anymore. however, it is unclear if this is the only solution, or if players can have other reasons to use or try different strategies. this might be connected back to the concept of meaningful decisions not being fully grasped yet. decoster and rubin [21] take this even further and state that people, players and spectators alike, would actually like imbalance and that games should generally be designed around fun, not balance. while there still can be balancing issues, those can mostly be offset by proper player matching, which should therefore be the focus. different authors also name various tools and methods to balance a game. those do not obviously exclude each other and contain, for example, user statistics, spread sheets, iterating, user feedback, power curves, tier lists and game theory, although the latter is more concerned with players than game design. paradigmplus (2020) 1:1 38 alexander becker and daniel görlich . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 conclusion & outlook our analysis of fourteen renowned authors’ concepts of game balancing revealed that their concepts are clearly different. although the authors usually agree on certain aspects, for example that games should provide meaningful decision options to the players, there is no central aspect, no central goal of balancing that all fourteen authors focused on in their respective books, texts, or videos. often, the authors are not far apart, though: for example, while novak, assuming the players’ perspective, concludes that a “game is balanced if players perceive that it is consistent, fair, and fun”, schell advises keeping the player in a state of continuous flow, and koster concludes that “game balancing aims at providing a good level of challenge for the user.” going into detail, it becomes obvious that some authors focus on certain types of games (e.g. sirlin and burgun on multiplayer games), while others try to address games in general (adams, schell, etc.). all of them, however, talk about games made for fun and entertainment, but not about games made for other purposes: serious games, health games, exergames and others that might need to be balanced towards achieving goals other than fun. still, the authors’ different perspectives already lead them to differing concepts about game balancing. our analysis, however, also revealed that never all, but at least most authors focus on four aspects of their respective game balancing concepts: the characteristics of well-balanced games, difficulty, symmetry, and the balancing process itself. combining these authors’ opinions on the characteristics of well-balanced games, meaningful decision options, player skill, and the prevention of dominant strategies seem to be pivotal. further combing their opinions on difficulty and symmetry, most authors seem to agree that finding the right level of difficulty is essential; that the difficulty should stay the same or grow steadily according to the player’s increasing skill; that fairness is inherent to symmetrical games but must be ensured in asymmetrical games alike; and that fairness alone does not make a game well-balanced. however, concerning the balancing process and the available means to balance games, the authors’ opinions and concepts differ. while many authors seem to strive to balance a game as well as possible, others argue for intentional imbalances and dynamic balancing, or even rating the players rather than balancing the game. though it is obvious that the balancing process is of utmost interest to renowned game designers wanting to share their knowledge and experience, their concepts about the balancing process itself are diffuse: they present many aspects and ideas, ranging from feedback loops and the transitivity of game elements to chance and metagame—but no author presented a practical, clear, and concise abstraction of an actual game balancing process. we therefore conclude that further research is required before a commonly agreeable definition of “game balancing” and a suitable abstraction of a practical game balancing process can be derived. it seems coherent that game balancing supports the design goal “fun”, especially by avoiding certain occurrences that ruin its fulfillment. however, we still lack a conclusive and positive definition of game balancing that is generally applicable, as well as the uttermost basic level of understanding of what all the rules and suggestions of implementation are supposed to achieve. not all games are necessarily designed primarily for fun and knowing what to avoid or how to tweak numbers is not sufficient to derive a full plan on how to design a game, for example. it may not be necessary to research game balancing that deeply, but we do not even know that for sure. an empirical study on how a good or bad game balance affects a game’s reception and success economically or in sports could show the importance and also help to find proven properties, but this might already require a deeper or more elaborated understanding than we currently have. acknowledgments the authors thank their colleague sarah hanagarth for proofreading and making this paper more easily readable. paradigmplus (2020) 1:1 what is game balancing 39 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . authors’ information – alexander becker studied philosophy with psychology at ruprecht-karls university heidelberg and virtual realities at srh university heidelberg. since then he has been researching game balancing and its underlying concepts. – daniel görlich studied informatics at humbolt university of berlin and holds a ph.d in humanmachine interaction from the university of kaiserslautern. currently, he is professor for virtual reality and game development at srh university heidelberg. his research focuses on modern and novel ways to design and develop virtual worlds. authors’ contributions – alexander becker prepared and conducted the semantic analysis of practitioners’ publications on game balancing, focused on the semantic analysis as well as on conceptual structuring, and wrote parts of the sections 2, 3, 4, 5, and 6 of the manuscript. – daniel görlich prepared and conducted the review of scientific approaches to game balancing, focused on discussion and interpretation of the results as well as on conceptual structuring, and wrote parts of the sections 1, 2, 3, 5, and 6 of the manuscript. competing interests the authors declare that they have no competing interests. funding no funding was received for this project. references [1] g. andrade, g. ramalho, a. gomes, and v. corruble, “dynamic game balancing: an evaluation of user satisfaction,” in john laird and jonathan schaeffer (eds.): proceedings of the second aaai conference on artificial intelligence and interactive digital entertainment (aiide 2006). aaai press 3–8, (marina del rey, california, usa), 2006. available at https://ojs.aaai.org/index.php/ aiide/article/view/18739. [2] v. volz, g. rudolph, and b. naujoks, “demonstrating the feasibility of automatic game balancing,” in tobias friedrich (eds.): proceedings of the genetic and evolutionary computation conference 2016, (denver, colorado, usa), pp. 269–276, 2016. available at https://doi.org/10.1145/ 2908812.2908913. [3] t. j. tijs, d. brokken, and w. a. ijsselsteijn, “dynamic game balancing by recognizing affect,” in second international conference on fun and games, (eindhoven, the netherlands), pp. 88–93, springer, 2008. available at https://link.springer.com/chapter/10.1007/ 978-3-540-88322-7_9. [4] g. van lankveld, p. spronck, h. van den herik, and m. rauterberg, “incongruity-based adaptive game balancing,” advances in computer games. acg 2009. lecture notes in computer science, vol. 6048, 2009. available at https://link.springer.com/chapter/10.1007/ 978-3-642-12993-3_19. paradigmplus (2020) 1:1 https://ojs.aaai.org/index.php/aiide/article/view/18739 https://ojs.aaai.org/index.php/aiide/article/view/18739 https://doi.org/10.1145/2908812.2908913 https://doi.org/10.1145/2908812.2908913 https://link.springer.com/chapter/10.1007/978-3-540-88322-7_9 https://link.springer.com/chapter/10.1007/978-3-540-88322-7_9 https://link.springer.com/chapter/10.1007/978-3-642-12993-3_19 https://link.springer.com/chapter/10.1007/978-3-642-12993-3_19 40 alexander becker and daniel görlich . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [5] r. wille, restructuring lattice theory: an approach based on hierarchies of concepts. technical university of darmstadt, 1981. available at https://link.springer.com/chapter/10.1007/ 978-3-642-01815-2_23. [6] u. priss, “formal concept analysis in information science,” in annual review of information science and technology, vol. 40, pp. 521–543, 2006. available at https://citeseerx.ist.psu.edu/ viewdoc/download?doi=10.1.1.60.4220&rep=rep1&type=pdf. [7] a. napoli and l. t. n. nguyen, “lattice editor.” available at https://latviz.loria.fr, 2016. accessed: 2019-08-09. [8] a. becker, game balance und ihre facetten – begriffsklärungen und konzept einer kategorisierung. srh university heidelberg, 2018. [9] e. adams, fundamentals of game design. new riders, 3rd ed., 2013. available at https:// proquest.tech.safaribooksonline.de/9780133435726. [10] a. rollings and e. adams, andrew rollings and ernest adams on game design. new riders, 1st ed., 2003. available at http://proquest.tech.safaribooksonline.de/1592730019. [11] k. burgun, “understanding balance in video games.” available at https://www.gamasutra. com/view/feature/134768/understanding_balance_in_video_.php, 2011. accessed: 201908-06. [12] d. sirlin, “balancing multiplayer games.” available at http://www.sirlin.net/articles/ balancing-multiplayer-games-part-1-definitions, 2002. accessed 2019-08-09. [13] j. portnow, “perfect imbalance – why unbalanced design creates balanced play.” available at https://youtu.be/e31osvzf77w, 2012. accessed: 2019-08-07. [14] d. felder, “design 101: balancing games.” available at https://www.gamasutra.com/blogs/ danfelder/20151012/251443/design_101_balancing_games.php, 2015. accessed: 2019-0811. [15] j. novak, game development essentials: an introduction. cengage learning, 2011. available at: https://www.amazon.com/game-development-essentials-jeannie-novak-ebook/ dp/b00b82g86o. [16] r. rouse and s. ogden, game design: theory and practice (wordware game developer’s library). jones & bartlett learning, 2nd ed., 2004. available at https://www.amazon.com/ game-design-practice-wordware-developers/dp/1556229127. [17] j. schell, the art of game design: a book of lenses. a k peters/crc press, 2nd ed., 2014. available at https://www.amazon.com/art-game-design-lenses-second/dp/1466598646. [18] i. schreiber, “game balance concepts. a continued experiment in game design and teaching.” available at http://gamebalanceconcepts.wordpress.com/, 2010. accessed: 2019-08-09. [19] t. sylvester, designing games: a guide to engineering experiences. o’reilly media, 2013. available at https://library.webknw.com/uploads/files/2020-10-07-08-36-21.pdf. [20] m. brown, “how games get balanced.” available at https://youtu.be/wxqzdxptb2a, 2019. accessed: 2020-02-26. [21] r. decoster and s. rubin, “pax south 2018 – balance in game design.” available at https: //www.youtube.com/watch?v=nxd8yq7j_qk, 2018. accessed: 2020-02-26. paradigmplus (2020) 1:1 https://link.springer.com/chapter/10.1007/978-3-642-01815-2_23 https://link.springer.com/chapter/10.1007/978-3-642-01815-2_23 https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.60.4220&rep=rep1&type=pdf https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.60.4220&rep=rep1&type=pdf https://latviz.loria.fr https://proquest.tech.safaribooksonline.de/9780133435726 https://proquest.tech.safaribooksonline.de/9780133435726 http://proquest.tech.safaribooksonline.de/1592730019 https://www.gamasutra.com/view/feature/134768/understanding_balance_in_video_.php https://www.gamasutra.com/view/feature/134768/understanding_balance_in_video_.php http://www.sirlin.net/articles/balancing-multiplayer-games-part-1-definitions http://www.sirlin.net/articles/balancing-multiplayer-games-part-1-definitions https://youtu.be/e31osvzf77w https://www.gamasutra.com/blogs/danfelder/20151012/251443/design_101_balancing_games.php https://www.gamasutra.com/blogs/danfelder/20151012/251443/design_101_balancing_games.php https://www.amazon.com/game-development-essentials-jeannie-novak-ebook/dp/b00b82g86o https://www.amazon.com/game-development-essentials-jeannie-novak-ebook/dp/b00b82g86o https://www.amazon.com/game-design-practice-wordware-developers/dp/1556229127 https://www.amazon.com/game-design-practice-wordware-developers/dp/1556229127 https://www.amazon.com/art-game-design-lenses-second/dp/1466598646 http://gamebalanceconcepts.wordpress.com/ https://library.webknw.com/uploads/files/2020-10-07-08-36-21.pdf https://youtu.be/wxqzdxptb2a https://www.youtube.com/watch?v=nxd8yq7j_qk https://www.youtube.com/watch?v=nxd8yq7j_qk what is game balancing 41 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [22] a. millard, “why are games so hard to balance?.” available at https://www.youtube.com/ watch?v=k3n-sy2ko4i, 2018. accessed: 2020-02-26. [23] r. koster, theory of fun for game design. paraglyph press, 2005. available at https://www. amazon.com/theory-game-design-raph-koster-ebook/dp/b004d4yi52. paradigmplus (2020) 1:1 https://www.youtube.com/watch?v=k3n-sy2ko4i https://www.youtube.com/watch?v=k3n-sy2ko4i https://www.amazon.com/theory-game-design-raph-koster-ebook/dp/b004d4yi52 https://www.amazon.com/theory-game-design-raph-koster-ebook/dp/b004d4yi52 introduction methodology state of the art semantic analysis for conceptual structuring discussion conclusion & outlook vol. 1, no. 2, pp 18 41, 2020 doi: 10.55969/paradigmplus.v1n2a2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . automated fine grained traceability links recovery between high level requirements and source code implementations alejandro velasco id 1,� and jairo aponte id 1 1universidad nacional de colombia, bogotá, colombia {savelascod, jhapontem}@unal.edu.co abstract software traceability has been a matter of discussion in the software engineering community for a long time. the process of keeping and recovering traces among software artifacts in any system represents a fundamental aspect to properly perform software maintenance tasks and requirements compliance verification. furthermore, there exist application contexts where this becomes a mandatory process, for instance, banking and healthcare. software traceability researchers have been proposing alternatives to recover lost traceability links in coarse-grained and middle-grained levels of detail; however, proposed techniques are not enough to meet the desired levels of granularity in specific critical contexts. in this work we propose a fine-grained traceability algorithm designed to recover traces between high level requirements written in natural language and source code statements where they are implemented. we tested our approach in four open-source healthcare systems to trace constraints requirements specified by the hipaa law, and evaluated the results as presented in this paper. keywords: software traceability ⋅ information retrieval ⋅ static code analysis ⋅ program slicing ⋅ software maintenance ⋅ natural language processing ⋅ healthcare received: 18 july 2020 ⋅ accepted: 13 august 2020 ⋅ published: 19 august 2020. 1 introduction traceability in software engineering is a research field whose main purpose is to recover lost information of traces and links between high-level artifacts (e.g., documentation, use case diagrams, requirements specifications) and source code artifacts (e.g., classes and methods) [1, 2]. throughout the evolution of any software system, the information of existing relations between requirements, source code artifacts, and underlying infrastructures may be lost, leading to a progressive increase in the complexity of software maintenance tasks. recovering those links among software artifacts constitutes a crucial requirement to ease code comprehension and help to perform a wide variety of tasks such as bug tracking, feature location, code restructuring, and change impact analysis. moreover, maintaining traceability links makes easier the validation of stakeholder’s needs and the verification of the correct implementation of requirements in any software system [3]. keeping the links among software artifacts is a task that software teams should do it from the very beginning of any software project [4]. however, in practice, in most cases, developers are focused on this article is distributed under the terms of the creative commons license attribution 4.0 international (cc by 4.0), which permits unrestricted use, distribution, and reproduction in any medium, providing appropriate credit to the original authors and source. https://doi.org/10.55969/paradigmplus.v1n2a2 https://orcid.org/0000-0002-4829-1017 https://orcid.org/0000-0002-5339-896x automated fine grained traceability links recovery between high level requirements and source code implementations 19 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . programming tasks and forget their responsibility of creating and maintaining documentation links between the various software artifacts. hence, when time permits, the system documentation, and some percentage of links recovery, is left for the final stage of the project [4]. these bad practices lead to significant drawbacks when assessing the quality of the constructed software products as well as the verification of their compliance with the mandatory constraints and stakeholders’ needs. in the case of critical systems, it is essential to comply with explicit privacy and security rules, as is the case of hipaa for healthcare information systems. software firms and regulatory entities, such as the hhs (u.s. department of health and human services) office for civil rights, are in charge of verifying and ensuring the regulatory compliance of software systems [5]. usually, regulators have no alternative but to perform manual checks of the source code to verify compliance with existing regulations [6, 7, 8]. such a way of working is a time-consuming and error-prone traceability task, and clearly, its main cause is the absence of traceability links. to aid people in mandatory constraints verification, researchers have proposed automated approaches to traceability links recovery. existing techniques focus mainly on linking classes and methods with high-level constraints and requirements. nevertheless, in some cases, the verification of compliance with mandatory rules requires analysis at a finer level of granularity to confirm and assure that source code implements them. fine-grained traceability link recovery aims at supporting the verification process of requirements compliance of software, by connecting mandatory constraints to concrete programming statements, including conditionals, assignments, loops, and basic code snippets. the main contribution of this paper is to present an extension of our previous work [9], proposing a fine-grained traceability technique that combines heuristics, information retrieval (ir) techniques, and static software code analysis, to identify traces between software mandatory constraints and source code structures. to address all the work presented in this paper, the following research questions were defined: 1○ what are the code structures commonly used to implement different types of mandatory constraints? 2○ how natural language processing, information retrieval, and static source code analysis techniques can support fine-grained traceability for mandatory constraints compliance? 3○ how can we assess the accuracy of a technique for fine-grained traceability link recovery? this paper is organized as follows: section 2 related work, briefly describes the studies relevant to our problem. section 3 dataset description, provides information about the artifacts used to conduct this study. section 4 heuristics, explains the observations and remarks we take as basis for the proposed technique. section 5 fine-grained traceability algorithm, presents the traceability technique designed and developed in this study. section 6 results, reports the evaluation data of the proposed technique. section 7 discussion, presents relevant remarks regarding the main strengths and weaknesses of the proposed approach. lastly, the paper ends with conclusions and future work, in sections 8 and 9, respectively. 2 related work requirements mapping is a time-consuming and error-prone task that is indispensable for ensuring that a software system complies with restrictions imposed by official regulations and constraints expected by stakeholders [7]. in the case of hipaa [10], several classifications by typology have been made to facilitate the validation process and make it more complete and comprehensive. hipaa law has three main sets of regulatory constraints; for healthcare software systems, hipaa establishes standards and rules aimed at ensuring security and confidentiality, and enforcing correct management of all patient data. [5]. in that sense, breaux. et al. [6, 11], propose a semantic model by identifying language patterns to extract rights and obligations from hipaa statutes; they considered the privacy rules on hipaa regulations for performing the requirements classification. the breaux model paradigmplus (2020) 1:2 20 alejandro velasco and jairo aponte . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . was the basis for the construction of other frameworks [12, 13, 14]. because the legal regulations are written in a complex and dense format, alshugran et al. proposed a process to extract the privacy requirements from hipaa; specifically, the defined process has a set of methods to analyze, extract and model privacy rules [15]. some studies have proposed mechanisms to encourage developers to verify legal compliance with hipaa requirements during the construction of healthcare software systems. for instance, maxwell et al. propose a production rule model to encourage requirements engineers to keep trace between law and high level artifacts across every development process [16]; they validated their approach using itrust, a hipaa compliant healthcare system. 2.1 traceability links recovery coarse-grained and middle-grained levels of granularity have been the most common in proposed approaches for traceability link recovery between high-level software artifacts and source code snippets [17, 18, 19, 20, 21, 22, 23, 24, 25]. fasano [18] built adams, a tool for automatic traceability management based on latent semantic indexing (lsi), that improves vector space model techniques [26, 27], under the assumption that most software systems have high-level software artifacts with a well defined hierarchical structure. adams is able to recover traceability links between software high-level documentation and source code classes. marcus and maletic [19] use latent semantic indexing (lsi) for recovering traceability links between methods and documentation. based on the comments and identifier names present in the source code, they manage to extract semantic information useful for recovering links. paloma et al. [23] created crystal (crowdsourcing reviews to support app evolution), an ir-based tool for recovering traceability links among commits, issues, and user reviews. the tool can remove useless words, calculate the textual similarity of the artifacts, and select the candidate links according to predefined criteria and threshold values. de lucia et al. [24] reduce the effects of noise in software artifacts by using smoothing filters. they empirically evaluated their approach and report a significant improvement of the ir techniques vector space model and latent semantic indexing. diaz et al. [25] propose tyrion, a tool for traceability link recovery using information retrieval and code ownership. they use code ownership to improve the candidate links generation. their main assumption is that if a developer authored code that is linked to a particular high-level artifact, then another code snippet created by the same author is likely to be associated with the same artifact. 2.2 fine-grained traceability link recovery techniques some techniques have been proposed that perform the recovery of traceability links at a fine-grained granularity level. this means that such techniques recover links between high-level artifacts and specific lines of code [28]. the execution traces of specified features of a software system, heuristics, and source code static analysis techniques have been used for this purpose [29, 30, 31, 8, 32]. wong et al. [32] come up with a technique based on execution slices, that receives two sets of test cases, one that exercises the feature of interest and another that does not. by using dynamic information obtained from the instrumented software system (i.e., list of statements executed), the technique can discriminate between code that is unique to a feature, and code that is common to several features. dagenais and robillard [30] propose a model to recover traceability links between the software learning resources and the application programming interface (api) documentation. they use a meta-model representation of the involved artifacts (i.e., documentation, source code, and support channels) to understand the context in which a code-like term is used. sharif and maletic [31] address the problem of evolving traceability links. for this purpose, they update and evolve traceability links based on the differentiation among versions of the involved artifacts in a traceability link. paradigmplus (2020) 1:2 automated fine grained traceability links recovery between high level requirements and source code implementations 21 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 limitations of adopted strategies as already mentioned, current ir-based techniques for recovering traceability links are designed to relate software requirements to files and classes. they have focused mostly on coarse-grained granularity, so that they usually trace high-level software artifacts to files, classes, and, in best cases, to methods. the main problem with fine-granularity source code structures, such as conditionals or exception handling statements, is that they usually do not contain enough textual information that matches the vocabulary of high-level artifacts [27]. this circumstance translates into a low accuracy of the ir-based techniques. therefore, new techniques or improvements of the existing ones should be able to generate traceability links between high-level documents, such as software mandatory constraints written in formal regulations, and low-level code structures such as conditionals, assignment statements, and method calls. 3 dataset description hipaa statutes are written in a complex language. thus, to trace source code artifacts back to highlevel constraints, we defined a taxonomy of hipaa statutes according to their likelihood of being related to software implementation of standards and regulations. moreover, we found that not all sections of the hipaa rules are related to healthcare management systems. thus, we decided to focus on the hipaa statutes most related to those systems and user data security regulations (as stated in the privacy rule). therefore, we extracted a set of law statutes that apply for our particular problem. table 1 summarizes the hipaa administrative simplification. in this study we analyze the security and privacy rule of hipaa since it includes most of the regulations related to the implementation of healthcare management systems. we consider those statutes derived from hipaa security and privacy rules, defined in the administrative simplification; then, we identify those that are suitable to associate with source code implementations and configurations in healthcare systems. as a result, we classified the law constraints using a taxonomy with three categories. the first category (a) has the rules that could be easily traced into code. these are regulations that mainly refer to functional requirements that must be implemented in every hipaa compliant healthcare system. the second category (b) includes statutes that define constraints related to software implementation and non-functional requirements, still suitable to be traced to code. the last category (c) lumps together statutes and standards that describe how organizations and health care providers implement practices and methods to audit interactions with healthcare systems so that the private information of patients is protected; those are unsuitable to be traced to code. table 1: hipaa statutes taxonomy id category subpart statutes samples a potentially suitable to trace into code. security standards: general rules. administrative safeguards. technical safeguards. 164.306(a)(1), 164.306(a)(2), 164.306(a)(3), 164.308(a)(1) (ii)(d), 164.308(a)(5)(ii)(c), 164.308(a)(5)(ii)(d). b moderately suitable to trace into code. administrative safeguards. 164.308(a)(1)(i), 164.308(a) (1)(ii), 164.308(a)(4)(ii)(a), 164.308(a)(4)(ii)(b). c indirectly related to health care software systems security standards: general rules. administrative safeguards. physical safeguards. 164.306(a)(1), 164.306(a)(2), 164.306(a)(3), 164.306(c), 164. 306(d), 164.306(d)(1),164.306 (d)(2), 164.306(d)(3). paradigmplus (2020) 1:2 22 alejandro velasco and jairo aponte . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 healthcare systems as a first step of the design of a fine-grained traceability algorithm, we look for java open-source healthcare software systems with available documentation (e.g. use cases and user stories). we found four candidate systems: itrust [33, 34], openmrs [35], oscar [36] and tapas [37]. the available documentation has suitable descriptions of the features offered by each system. moreover, these four health care systems are explicitly defined as hipaa compliant in their official pages. table 2 shows the repository and system version analysed in this study. table 2: summary of open source health care systems chosen to analyze in this study. name repository version tapas https://sourceforge.net/projects/ap-apps/ v. 0.1 itrust https://sourceforge.net/projects/itrust/ v. 21 oscar https://sourceforge.net/projects/oscarmcmaster/ v. 14.0.0 openmrs https://github.com/openmrs v. 1.12.x each health care system offers features that implement standards defined in hipaa law. the features that we identified in all systems, by observing the deployed applications, include user authentication, patient historical data management, appointment definitions, disease reports, audit controls, and data encryption. for each system, we identified the file extensions associated with source code artifacts, as showed in table 3. for source code analysis we consider .java .js and .jsp extensions. such files have java and javascript statements, and also html and custom tags; the .xml and .properties extensions correspond to files with relevant information about constants and variables used within the source code, and therefore, are also pertinent for our purpose. lastly, files with .sql extension has sql sentences with significant information about data and entities of the application domain that are referenced in the source code. table 3: summary of source code files containing source code artifacts that were taken as the corpus of the traceability algorithm. itrust openmrs oscar tapas java 936 1545 3810 221 jsp 266 447 1655 0 js 3636 416 875 0 xml 14 820 193 14 sql 192 2 711 0 props 7 39 39 2 total sc artifacts 5051 3269 7283 237 3.2 extracting requirements from high level artifacts one of the data items that we require for this study is explicit high-level requirements. to this end, we had to extract specific requirements from the available documentation on each health care system. we set apart several text fragments from requirements explicitly declared in the software highlevel artifacts (query of our algorithm). then, we extracted software requirements embedded in sentences from the documentation of all four systems, according to the following template 1: [article]+ [subject]+ [obligationv erb]+ [complement] (1) paradigmplus (2020) 1:2 automated fine grained traceability links recovery between high level requirements and source code implementations 23 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 4: some specific requirements extracted from the available documentation of selected healthcare systems id requirement system documentation itrust-1 "an hcp is able to create a patient [s1] or disable a selected patient [s2]. the create/disable patients and hcp transaction is logged (uc5)." itrust http://agile.csc.ncsu.edu/ itrust/wiki/doku.php? id=requirement oscar-1 "your password is stored in an encrypted format such that even the system administrator cannot find out what it is. if you have forgotten your user name and/or password, your administrator can reset the password for you but he/she cannot tell you what the original password was" oscar http://oscarmanual.org/ oscar_emr_12/general %20operation/accesspreferences-andsecurity/accessing-oscar tapas-1 "the system must have the ability to manage users in the system. like clinical data, users should not be able to be deleted from the system as they will be tied to activities in a record." tapas http://tap-apps.source forge.net/docs/usecases.html#ucsysadm-01 thus, the extracted requirements were textually taken from documentation without any modification. furthermore, the extraction process was conducted only from available use cases and user stories. those artifacts convey information about requirements and user needs that is essential for subsequent implementation tasks. a user story describes functionality that will be valuable to any actor, ranging from software developers to final users [38]; therefore, it is likely probable to find implementations in source code based on textual information from user stories. by the same token, use cases typically describe interactions between the system and its users that allow them to achieve their goals. thus, they also represent a mechanism to define software requirements that somehow will be represented in source code. in some cases, following the precise definition of the format leads to the extraction of phrases with no sense or context. for that reason, in such particular cases, we included in the extraction additional information from the surrounding text of such phrases to complete them, i.e., terms that were not considered when the filter was applied. table 4 shows some examples of specific requirements extracted from artifacts of each system. finally, we selected all the extracted requirements that were more suitable to trace into code according to the proposed taxonomy of hipaa security and privacy regulations. in other words, we associated each extracted requirement with a hipaa statute and focused on those in the categories (a) and (b) of the classification in table 1. 3.3 source code extraction before proposing a traceability algorithm to link up requirements and source code structures, we study in detail how different software healthcare systems are built. the intent of this process was to identify patterns in the source code and come up with a solution based on empirical observations. in this section we describe the examination process conducted through the source code of the selected healthcare systems presented in table 2. for each of the requirements selected from all the systems (those that were related to any hipaa statute), we identified files and source code lines that played a dominant role in their implementation. this process required a joint effort that involved several people with knowledge of the programming paradigmplus (2020) 1:2 24 alejandro velasco and jairo aponte . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 1: overview of source package organization for each of the healthcare systems selected in the this study. on left top, itrust source packages, right-top tapas source packages, bottom-left openmrs source packages, bottom-right oscar source packages. languages used in the implementation of the healthcare systems. once deployed and configured the development environments for the four healthcare systems according to the specifications given in user manuals and deployment instructions, we performed the analysis of the source code. in general terms, given a requirement, we conducted a process of exploration starting from the layer of views, followed by the layer of services until the layer of entities at the database level. figure 1 shows an overview of the source code organization from the used ides for each of the projects. for all the selected requirements extracted from documentation, we collected detailed information about the source code files used in the implementation, the methods that were defined in those files and also the strictly related code lines within each method. for each trace of source code, we register the information to be processed later, when evaluating the effectiveness of the proposed traceability algorithm. 4 heuristics over time, the solutions to complex problems in software engineering have often been based on premises that greatly simplify the search for an answer. these premises can be derived from the diparadigmplus (2020) 1:2 automated fine grained traceability links recovery between high level requirements and source code implementations 25 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . rect observation of a problem in such a way that through an empirical process, the limits and starting points of a possible solution can be defined. heuristics are often used in multiple contexts to propose possible solutions to problems of diverse characteristics, such as design patterns [39], computational problems[40], optimization algorithms [41] and fault location in software [42]. in this context, we denote the term heuristic as a problem-solving approach used to improve the accuracy of our solution. due to the nature of the problem that we tackle, we propose some heuristics to reach an approximate solution to find traceability links between artifacts whose domains are quite dissimilar, i.e., source code and natural language documents. based on observations derived from the manual process of traceability links recovering between the source code and the high level requirements described in natural language, we defined a set of heuristics that determined the basis for the design of an automated fine-grained traceability algorithm. 4.1 (h1) heuristic 1: developers tend to use terms present in high-level requirements when writing source code before giving a solution to a specific domain problem, any software development process goes through a phase of abstraction in the first place. abstracting a problem requires an understanding of all the business rules, entities and interactions between the different components. the abstraction process requires identifying the entities that interact within a problem domain. to avoid losing the context of the problem when creating a solution, software architects and developers often naturally use descriptive terms to generate the software artifacts needed for the construction of a system. for instance, some programming paradigms such as object oriented programming [43] and aspects oriented programmings [44] are based on abstraction processes to identify actors and key entities in a determined context; once such key entities have been identified, as well as the business rules present in the domain, it is feasible to implement a suitable solution that models the real situation. 4.2 (h2) heuristic 2: comments in source code are a valuable source of information that can be exploited to increase the accuracy of any traceability algorithm in addition to the information encoded in the source code identifiers, developers working on open source projects often write comments to explain functionalities that are implemented at the source code level [45, 46]. making comments in the code is a practice that could be reduced as long as good coding techniques are followed. in high-level programming languages, the source code itself should be sufficiently explanatory so that no additional comments or explanations are required. [47]. in spite of that, in the health care management systems selected for conducting this study, the frequency of comments and explanatory texts within the code is very high; comments range from explanations about the implemented code to rudeness or bad words. while it is true that the presence of comments in the source code can improve the accuracy of a traceability algorithm based on textual similarity, it must be kept in mind that not all the comments in the source code provide a significant input. in some cases, stated information is not related at all with the requirements that are being implemented. 4.3 (h3) heuristic 3: subjects in texts that describe a high-level software requirement regularly represent entities in the source code given a software requirement written in natural language, there is often a subject, a verb, and an object of determined action. for that reason and as a consequence of h1, when developers are implementing paradigmplus (2020) 1:2 26 alejandro velasco and jairo aponte . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . a requirement at the source code level, entities, methods, and variables names usually include terms directly associated with the subjects and objects described in a high-level requirement. as our problem domain is health care, there is a great variety of terminology that if not adopted when a requirement is implemented, it would be hard and tedious to understand the code. in the particular context of healthcare management systems, terms like “patient”, "appointment", "disease" and "health care provider" are usually present in a large number of artifacts. 4.4 (h4) heuristic 4: conventions used by developers to name source code entities can be used to define patterns to improve accuracy in ir algorithms conventions in the source code are a mechanism accepted by the vast majority of developers to conduct an specific development process [48, 49]. nowadays, there are many frameworks that prioritize convention over configuration allowing developers to focus on the comprehension of the problem domain and business rules that must be implemented, instead of tedious and repetitive technical details. for instance, frameworks such as ruby on rails and groovy on grails provide a big set of code artefacts tested by the community; there is a well-known collection of tools that allow developers to assume conventions and reuse tested artefacts in the development process leaving aside technical details, in other words, developers trust the functional code tested by the community and focus their efforts on the problem domain. although the health care systems that we analyzed in this study were not built over the set of tools offered by a framework such as ruby on rails, the adopted conventions that we managed to observe after a detailed analysis of the source code include ways to name the different code artifacts (e.g. camelcase, uppercamelcase), design patterns, folder structures, and code assets organization. well defined conventions are present in projects where a large number of people work together (e.g. health care systems selected); in that way it is more addressable to articulate teamwork and facilitate readability of the source code. in open source projects, the community usually defines its own conventions. 4.5 (h5) heuristic 5: delegation is ubiquitous in object-oriented systems and can be used to determine the artifacts that are involved in the implementation of a high-level requirement delegation is a concept that can be extrapolated in any context since every single part of any system can assign responsibilities to another one perform a particular task. delegation in source code is a design pattern where different objects that make up a system, expose behaviors in such a way that the responsibility to perform a certain operation does not fall on a single member but is assigned to different instances and in this way, the work is distributed [47]. the software delegation facilitates the structuring of the source code and facilitates software maintenance tasks. delegation in general is always present when the architecture of a system is defined. this principle is important and should not be overlooked to propose a solution to the traceability problem. since different members in the source code delegate responsibilities, it would be wrong to assume that the implementation of a high level requirement will be condensed in a single member or component within the system. as a consequence of this premise, in the context of our problem, it is necessary to support any information retrieval (ir) process based on textual similarity with techniques of static code analysis to determine delegation interactions between members that may be involved in the implementation of a high-level requirement. paradigmplus (2020) 1:2 automated fine grained traceability links recovery between high level requirements and source code implementations 27 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 fine-grained traceability algorithm a great variety of techniques have been formulated to perform traceability recovery processes in software artefacts. in order to verify compliance of regulatory constraints in a fine granular level, we proposed an algorithm that take advantage of information retrieval (ir) techniques, static code analysis and heuristics derived from observations after a process of manual analysis over source code implementations. figure 1, shows a summary of our technique. figure 2: flow diagram of proposed fine-grained traceability algorithm. 5.1 a1: call graph construction from source code files the source code files and artifacts and relevant structures that are part of the implementation of a system are all of a quite different nature. java classes are constituted by attributes and methods; javascript code files are batch processing files that are stored in plain text; jsp files are a combination of java code, javascript code, tomahawk and html tags. the process of a1 as depicted in figure 2, can be summarized as the generation of a plain text file with the information of methods headers (i.e. access modifier [optional], return type, name, parameter list, exception throws [optional]) and their paths in the different source code artifacts. figure 3, shows more detailed view of this file. 5.2 a2: indexing phase before going through the algorithm of information retrieval, several processing tasks are performed on the text for both the corpus and the query. the query is a list of requirements writing in natural language to be traced into the corpus, which is the source code of a selected system. when dealing with two entries of a different nature, the treatment of the terms that make up the corpus and the query is essentially different in both cases. concerning the treatment performed on the paradigmplus (2020) 1:2 28 alejandro velasco and jairo aponte . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 3: first 15 lines of the method mapping file for itrust query, tokenization, stemming, stopword removal, part of speech (a.k.a. pos) tagging, punctuation removal, number spell out and word embedding [50] tasks are all performed. on the other hand, the tasks carried out with the terms that make up the corpus include stop word removal, splitting by camel case, punctuation removal and tokenization. figures 4 and 5 show examples of corpus and query files after the indexing phase. 5.3 b: information retrieval process once the corpus and the query are indexed, they are treated as entries for the information retrieval algorithm to obtain a ranked list of the possible corpus files that are most likely related to the text of the query requirements. figure 4: fragment of query file for itrust paradigmplus (2020) 1:2 automated fine grained traceability links recovery between high level requirements and source code implementations 29 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 5: fragment of corpus file for itrust after executing the information retrieval algorithm, k files are obtained in order of relevance for each requirement; that is, if the query file contains a set of n requirements in total, the list of ranked links will contain a total of kn files most related to each entry in the query file. the number of links taken into account (k) for each requirement was determined empirically. in other words, after 20 executions of fgthunter over the selected systems (i.e. 5 evaluations for each selected system) we determined that the optimal value for k fluctuates between 25 and 30 files per requirement. this number is enough to obtain the most strictly relevant files, ignoring those that are less related to requirements. an example of the ranked list of links given by the traceability algorithm are shown in figure 6. figure 6: first 7 results in the ranked list of artifacts generated for itrust after applying lsi algorithm. 5.4 c: parsing phase once the list of files, most related to a given requirement based on textual similarity is obtained, the next phase of the algorithm consists of analyzing source code implementations and perform abstractions through program slicing, to find structures at source code level that are related to the implementation of a particular requirement. to correctly extract different structures within the code, we use paradigmplus (2020) 1:2 30 alejandro velasco and jairo aponte . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . parsers for each type of file included in the corpus file. the parsers statically analyze code elements present in .java, .js, .jsp, .xml, .sql, .properties and plain text files. parsers receive as input a source code fragment and give us an abstract syntax tree (ast) representation of it in a way that makes it easier to process. 5.5 d: program slicing process some mechanisms at source code level have been proposed to support traditional optimization techniques. one of those mechanisms is program slicing; this abstraction technique has multiple applications in software engineering such as testing [51], refactoring of the source code, reverse engineering, software comprehension and software maintenance [52]. formally, we can define program slicing as an abstraction process [53] that takes place at the code level, with the purpose of looking for fragments (slices) of code that directly or indirectly affect the state of a variable (slicing criteria); in this way it is possible to simplify the source code of the programs focusing on previously selected semantic or syntactic aspects. the slicing process eliminates all those parts of the code that are not related or that have no effect on the criterion of interest [54]. after executing the parsers for each type of source code file, it is possible to extract code fragments of interest since we are able to perform an exhaustive static analysis of source code. for each of the files that were identified by the information retrieval algorithm (lsi), slicing criteria were defined considering the subjects present in the description of each requirement. within the source code, there are terms (e.g. subjects, adjectives, verbs) of the problem context and therefore it is possible to define slicing criteria from the names of variables, objects, classes, and methods. to illustrate this point, let’s assume that a requirement r1 contains 4 subject names; so it is possible to define 4 slicing criteria to conduct a program slicing process in each file given by the retrieval information algorithm for r1 after executing the process defined in b (figure 2). for each source code file, a slicing process is performed to get the code lines within each file that affect or are affected by a particular criterion. the result of this operation is a set of slices for each file ranked in b. 5.6 e: full trace construction phase the information given by the process of abstraction of the source code through program slicing is not enough to build entirely the trace of a requirement. to put it another way, consider a slice of source code given by the process defined in d; it is highly likely that such a code snippet contains calls to other methods within the system that do not necessarily contain terms in common with a high-level requirement but that fulfill a very important role in the implementation. this problem can also be seen as a consequence of delegation. in this phase of the algorithm, the code snippets obtained in the slices are analyzed and, for each call to an external method that is not defined in the parent file, a search process is carried out according to the definition header of the method that is invoked with the support of the mapping file obtained in a1. each of the referenced methods is also included in the trace of the requirement, and recursively, a slicing process d is performed. as a final result of the execution of the traceability algorithm for a particular project, a list of files and source code structures that are most related to each of the requirements specified in the query file are obtained. figures 7 and 8 show an example of the expected output after the full traceability algorithm execution. to evaluate and test our approach, we developed a java program that implements the proposed traceability technique. it was named fine grained traceability hunter (fgthunter). we also implemented tools designed for static analysis of source code, as well as information retrieval algorithms. paradigmplus (2020) 1:2 automated fine grained traceability links recovery between high level requirements and source code implementations 31 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . figure 7: folder structure of results obtained after the execution of the traceability algorithm for itrust. parent folder contains stats about the algorithm execution, corpus and query files, method mapping file and result summary file. results folder contains subfolders for each requirement specified in the query file, containing the most related artefacts and source code lines for a given requirement. figure 8: example of the content for a particular result file. the source code slices and method calls are all included in the trace. 6 results we executed the traceability algorithm implemented in the fgthunter tool for each of the four open source systems. we collected the results and evaluated the precision of our technique to find traceability links between high level artifacts and source code lines. the algorithm was able to successfully find a large number of the high-level requirements filtered for this study, along with new traceability links that were not originally found by the manual construction of traces. the results were grouped for each of the analyzed systems. table 5 summarises the findings for trust system; table 6 shows the collected results for openmrs, table 7 presents the results for tapas and table 8 collects the results for oscar. to evaluate the performance of our technique, we calculated the f1 score for each extracted requirement that was associated with a hipaa statute; then, we calculated the average of f1 scores obtained in each statute. results were grouped for each of the analyzed systems. table 9 summarizes our findings. provided that, it is important to denote that for a given hipaa statute, not always every paradigmplus (2020) 1:2 32 alejandro velasco and jairo aponte . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 5: fgthunter results for itrust requirements. r-id totallinks true positives true positives not found gold set precision recall f1 1 706 265 441 0 4 0,38 1 0,55 3 341 149 192 0 24 0,44 1 0,61 4 411 50 361 17 22 0,12 0,75 0,21 5 382 18 364 1 1 0,05 0,95 0,09 21 306 62 244 6 48 0,2 0,63 0,31 24 25 12 13 4 8 0,48 0,71 0,57 25 97 26 71 9 33 0,27 0,76 0,4 37 121 35 86 0 1 0,29 1 0,45 38 120 31 89 0 1 0,26 1 0,41 40 297 116 181 0 6 0,39 0,98 0,56 45 514 161 353 0 28 0,31 0,98 0,47 47 411 165 246 1 12 0,4 1,01 0,57 110 165 79 86 3 9 0,48 0,99 0,64 111 164 79 85 3 8 0,48 1 0,65 143 906 14 892 22 22 0,02 0,39 0,03 144 498 170 328 3 30 0,34 0,87 0,49 145 166 43 123 0 7 0,26 1 0,41 163 222 41 181 0 9 0,18 1 0,31 164 456 12 444 9 10 0,03 0,57 0,05 187 120 97 23 0 1 0,81 1 0,89 189 130 67 63 0 5 0,52 0,93 0,66 191 229 3 226 0 1 0,01 0,75 0,03 180 74 36 38 0 1 0,49 1,13 0,68 selected system had at least one related high level requirement; that explain why the value of column "average f1 score" was not calculated in all the four system in each hipaa statute. the audit management is a fundamental part of any software system that aims to keep track of all the operations performed by every actor that use the system. table 9 shows the average of the harmonic mean (average of f1 score) for the requirements in each healthcare system that are related to hipaa statute 164.312 (b). the precision of fgthunter for itrust and oscar systems had an approximate value of 0.4 whereas in tapas the precision was much lower. when performing the manual code inspection, we noticed that very few code lines in tapas handled the audit control; this fact may explain the low precision in this case. the access control to the information within a system ensures the correct manipulation of the data. in the four systems that we analyzed, the access to the information was restricted by the definition of roles and user permissions to view, read, modify, or remove data. table 9 shows the f1 score means of fgthunter obtained when using it for finding the lines of code most related to the requirements table 6: fgthunter results for openmrs requirements. r-id linksretrieved true positives true positives not found gold set precision recall f1 53 1338 1041 297 0 11 0,78 1 0,88 80 428 226 202 10 33 0,53 0,89 0,66 paradigmplus (2020) 1:2 automated fine grained traceability links recovery between high level requirements and source code implementations 33 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 7: fgthunter results for tapas requirements. r-id linksretrieved true positives true positives not found gold set precision recall f1 1 939 179 760 11 11 0,19 0,94 0,32 3 248 77 171 12 19 0,31 0,87 0,46 21 872 393 479 8 14 0,45 0,98 0,62 22 479 179 300 10 14 0,37 0,95 0,54 42 16 1 15 10 10 0,06 0,1 0,08 43 647 17 630 10 10 0,03 0,63 0,05 44 139 30 109 4 4 0,22 0,88 0,35 51 20 3 17 10 10 0,15 0,3 0,2 54 4610 2893 1717 0 12 0,63 1 0,77 56 16 1 15 10 10 0,06 0,1 0,08 57 1231 452 779 0 13 0,37 1 0,54 59 16 1 15 10 10 0,06 0,1 0,08 60 3608 1465 2143 0 20 0,41 1 0,58 62 16 1 15 10 10 0,06 0,1 0,08 70 3236 1337 1899 0 10 0,41 1 0,58 72 343 8 335 7 12 0,02 0,53 0,04 73 16 1 15 10 10 0,06 0,1 0,08 75 288 134 154 0 10 0,47 1 0,64 77 336 89 247 8 12 0,26 0,92 0,41 81 336 89 247 8 12 0,26 0,92 0,41 associated with the statute 164.312 (a) (1) of hipaa in each system. the average harmonic mean has an acceptable level, greater than 0.4 points, in openmrs, oscar and tapas. on the other hand, for itrust the algorithm had the lowest performance. what happens is that the access control by roles in itrust was handled by access restrictions defined in the configuration .xml files for the tomcat server. table 8: fgthunter results for oscar requirements. r-id linksretrieved true positives true positives not found gold set precision recall f1 1 8440 605 239 12 34 0,72 0,98 0,83 2 691 531 160 2 12 0,77 1 0,87 3 876 775 101 3 28 0,88 1 0,94 5 380 136 244 12 22 0,36 0,92 0,52 40 1777 853 924 8 102 0,48 0,99 0,65 44 1110 636 474 22 92 0,57 0,97 0,72 45 1216 1065 151 10 203 0,88 0,99 0,93 46 1273 317 956 31 43 0,25 0,91 0,39 47 262 236 26 0 34 0,9 1 0,95 49 6575 6012 563 0 106 0,91 1 0,96 56 3969 949 3020 52 52 0,24 0,95 0,38 63 142 28 114 7 9 0,2 0,8 0,32 70 1668 462 1206 2 2 0,28 1 0,43 paradigmplus (2020) 1:2 34 alejandro velasco and jairo aponte . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . table 9: summary of results for all healthcare systems analyzed in this study. hipaa statute description average f1 score 164.312(b) “implement hardware, software, and/or procedural mechanisms that record and examine activity in information systems that contain or use electronic protected health information.” itrust:0,39 oscar:0,39 tapas:0,15 164.312(a)(1) “implement technical policies and procedures for electronic information systems that maintain electronic protected health information to allow access only to those persons or software programs that have been granted access rights as specified in §164.308(a)(4).” itrust:0,25 oscar:0,39 openmrs:0,66 tapas:0,39 164.312(a)(2) (i) “assign a unique name and/or number for identifying and tracking user identity.” itrust:0,43 oscar:0,65 openmrs:0,77 164.312(d) “implement procedures to verify that a person or entity seeking access to electronic protected health information is the one claimed.” itrust:0,60 oscar:0,87 tapas:0,46 164.312(a)(2) (iii) “implement electronic procedures that terminate an electronic session after a predetermined time of inactivity.” itrust:0,30 oscar:0,32 tapas:0,46 164.308(a)(5) (ii)(c) “procedures for monitoring log-in at tempts and reporting discrepancies.” itrust:0,48 oscar:0,73 openmrs:0,66 164.308(a)(1) (ii)(d) “implement procedures to regularly review records of information system activity, such as audit logs, access reports,and security incident tracking reports.” itrust:0,40 oscar:0,39 164.312(e)(2) (i) “implement security measures to ensure that electronically transmitted el ctronic protected health information is not improperly modified without detection until disposed of.” itrust:0,41 164.308(a)(5) (ii)(d) “procedures for creating, changing, and safeguarding passwords.” itrust:0,53 oscar:0,69 164.312(a)(2) (iv) “implement a mechanism to encrypt and decrypt electronic protected health information.” oscar:0,83 164.308(a)(7) (ii)(a) “establish and implement procedures to create and maintain retrievable exact copies of electronic protected health information.” oscar:0,96 tapas:0,58 164.308(a)(4) (i) “establish and implement procedures to create and maintain retrievable exact copies of electronic protected health information.” tapas:0,32 164.312(c)(2) “implement electronic mechanisms to corroborate that electronic protected health information has not been altered or destroyed in an unauthorized manner.” tapas:0,63 164.312(c)(1) “implement policies and procedures to protect electronic protected health information from improper alteration or destruction.” tapas:0,51 assigning a unique identifier to each entity of a system ensures the correct manipulation of the data and facilitates the control of the information integrity. according to our observations, the reparadigmplus (2020) 1:2 automated fine grained traceability links recovery between high level requirements and source code implementations 35 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . strictions of non-repetition were always defined at the database level, that is, the traceability of the requirements associated with the statute 164.312 (a)(2) (i) of hipaa was carried out analyzing .sql files with thousands of code lines impacting negatively on the precision of our technique. however, according to the results, for oscar and openmrs systems the algorithm reaches its highest performance, probably because at the application level, the uniqueness of the identifiers for each entity was also validated. ensuring the correct implementation of access control mechanisms is a fundamental aspect in any system that manipulates critical information. such control strategies range from verification of passwords and access codes to role management within the system. considering the wide spectrum of artifacts in each system that may be related to hipaa statute 164.312 (d), it is natural that the accuracy tend to be very high; however in tapas that value is low, maybe because the access control in this application is not clearly defined at the application level, i.e., there are no login forms or access restrictions, and externally they must control the access to the information by applying restrictions policies. closing the session after a period of inactivity is a measure of additional protection that usually exists to avoid improper manipulation of the data. the session time limit for a given user is often specified in a particular line of code within the application, either through a property file, a database record or a global variable. considering the quantity of artifacts analyzed and the reduced number of code lines in which the requirements related to statute 164.312 (a)(2)(iii) of hipaa are implemented, for itrust and oscar the level of precision was very low. according to table 9, tapas is the exception probably because in many parts of the code the mechanisms for closing inactive sessions are repeated, once a particular operation has started. as a security measure, many information systems keep a record of unsuccessful attempts prior to login for a particular service. depending on the number of failed attempts, an account is blocked for a period of time or indefinitely until an administrator decides to unblock it. table 9 summarizes the traceability results for the requirements related to hipaa statute 164.308 (a)(5)(ii)(c). the case with less precision was itrust, probably because there are very few code lines where the control of failed attempts is made. when a user of the system views or edits the information of a particular patient, the audit information should be enough to know the detail of the modifications made to critical data, as is established by the statute 164.308 (a)(1)(ii)(d) of hipaa. table 9 shows the average of the harmonic mean for the systems that implemented control mechanisms of visualization for audited data. in general, the performance of the traceability algorithm was acceptable, above 0.4. as a measure of securing information in the event of a disaster, data security should be periodically generated from the information contained in the database in order to be effectively restored. the hipaa statute 164.308 (a)(7)(ii)(a) refers to this topic, as shown in table 9. oscar was the system in which our algorithm reached a higher precision, while tapas was the opposite case. this particular behavior can be proved when we compare the source code artifacts from both systems that were involved in the implementation of such a statute. from our findings in the preliminary analysis, we observe that oscar has more lines of code associated with data backups and database restoration than tapas. 7 discussion this paper presents a novel approach to a fine-grained traceability problem that is common in the healthcare systems context. our findings indicate that the precision of fgthunter is, in general, more than 0,4 for each hipaa requirement (average f1 score). the regulations related to audit control standards and session expiration in the implementation of healthcare systems were the hardest to trace to source code statements. in those cases, our approach achieved the lowest precision (about 0,35 f1 score mean, for each system). very few lines and source paradigmplus (2020) 1:2 36 alejandro velasco and jairo aponte . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . code structures related to these requirements were successfully mapped by our algorithm for each system. this can be explained by the few places within the source code where these requirements were implemented, increasing in that way the recall of our technique and decreasing the precision. with respect to the other statutes defined in the privacy and security rule of hipaa, the performance of our algorithm was acceptable (over 0,4 f1 score mean) and in some cases excellent (over 0,8 f1 score mean). the precision of the proposed technique is highly dependent on good design practices and the adherence to conventions for naming entities according to the problem domain (i.e., quality of source code); this is a key factor that should be considered in future research to increase the accuracy of any text-based technique. high level requirement extraction from available resources following a linguistic pattern, may compromise the correctness and completeness of query artifacts in our algorithm, since not always documentation written in natural language is good enough to describe what actually is implemented in source code. to reduce the impact of this issue, a meta-language could be proposed to denote high level requirements prior the definition of query artifacts. although it is not a mandatory task in the execution of our technique, when starting from requirements that are defined from a specific context (i.e. system domain) the precision of the technique would improve. 8 conclusions in specific contexts, as is the case of medical health systems, strict rules are required to comply with the secure processing of personal data. for the effective verification of compliance with these rules, it is essential to establish links between the formal text of the regulations and the lines of code in which they are implemented. in this paper we describe an algorithm, based on heuristics, static code analysis and information retrieval techniques, that supports the recovering such traceability links. one of the biggest difficulties encountered is that the regulations expressed in legal documents are usually written in fairly technical language and detached from a particular context. to overcome this problem, we manually identified and extracted the documentation of requirements that is related to some hipaa statute, within the taxonomy that we previously defined. even though the static code analysis techniques have played a determining role in the creation of automatic techniques for the recovery of traceability links, our approach properly combines those techniques with heuristics and information retrieval methods. we believe that these combined methods can improve the results of the individually used techniques, and could reach an adequate level of precision to automatically retrieve the links between formal regulations and source code. although it is quite difficult to reach a suitable degree of precision in a traceability technique based on an information retrieval algorithms, without the assistance of a person that helps to delimit the list of ranked candidate links, our approach would undoubtedly ease the work of those people involved in the process of certifying that specified system complies with rules and standards at a source code level. in any case, further empirical validations are required to assess the effectiveness of our approach in real contexts. 9 future work our work can be expanded in many directions. first, as the technique is designed for software systems written in java ee, future work could focus on expanding the current traceability algorithm to cover other languages and systems development environments. for this purpose, one alternative would be to keep the original approach of our technique and build syntactic analyzers for each new programming language in such a way that the source code slicer and code analysis algorithms can be applied as they are. paradigmplus (2020) 1:2 automated fine grained traceability links recovery between high level requirements and source code implementations 37 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . second, another evaluation of our technique can be made with other regulations other than hipaa. to do this it would be essential to analyze the structure of such regulations and identify appropriate software systems whose documentation and code are available to carry out the entire process outlined in this paper. lastly and most importantly, empirical evaluations of our approach are more than desirable. they are required to assess the impact of our technique on the work of real users and regulation reviewers as well as its performance, usability, and compliance with real needs. authors’ information – alejandro velasco is a software engineering practitioner currently working in scotiabank colpatria. he received his engineering and master’s degree in systems and computing engineering from universidad nacional de colombia (bogotá, colombia). his research interest includes software engineering, chaos engineering and machine learning. – jairo aponte is an associate professor in the department of computing systems and industrial engineering at universidad nacional de colombia. he received his engineering and a master’s degree in computing systems engineering from universidad de los andes (bogotá, colombia), and a ph.d. in engineering from universidad nacional de colombia (bogotá). his research interests include agile software processes, program comprehension, and software evolution. authors’ contributions – alejandro velasco conducted the analysis of hipaa statutes and proposed the derived taxonomy, built the dataset and goldset from open source heatlh care systems as presented in section 3, proposed the set of heuristics based on observations as described in section 4, designed the fine grained traceability algorithm presented in section 5, implemented the proposed technique by creating fgthunter, tested the proposed approach and described the results as stated in section 6; and finally, conducted the analysis of results and discussions presented in sections 7, 8, and 9. – jairo aponte validated the constructed dataset presented in section 3, validated and improved the proposed heuristics presented in section 4, participated in the design and improved the performance of the proposed algorithm by suggesting changes based on previous research, improved the analysis of results described in section 6; and finally, participated in the writing of all sections. competing interests the authors declare that they have no competing interests. funding no funding was received for this project. availability of data and material 1○ datasets and goldsets for each of the systems described in this work, as well as hipaa artifacts can be found in the next repository for public access: https://bitbucket.org/savelascod/ fgt-attachments 2○ source code files of fgthunter can be found in the next repository: https://bitbucket.org/ savelascod/fgthunter paradigmplus (2020) 1:2 https://bitbucket.org/savelascod/fgt-attachments https://bitbucket.org/savelascod/fgt-attachments https://bitbucket.org/savelascod/fgthunter https://bitbucket.org/savelascod/fgthunter 38 alejandro velasco and jairo aponte . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . references [1] o. c. z. gotel and c. w. finkelstein, “an analysis of the requirements traceability problem,” in proceedings of ieee international conference on requirements engineering, pp. 94–101, 1994. [2] g. antoniol, g. canfora, g. casazza, a. de lucia, and e. merlo, “recovering traceability links between code and documentation,” ieee transactions on software engineering, vol. 28, no. 10, pp. 970–983, 2002. [3] b. ramesh and m. jarke, “toward reference models for requirements traceability,” ieee transactions on software engineering, vol. 27, no. 1, pp. 58–93, 2001. [4] v. yadav, r. k. joshi, and s. ling, “evolution traceability roadmap for business processes,” in proceedings of the 12th innovations on software engineering conference (formerly known as india software engineering conference), isec’19, (new york, ny, usa), association for computing machinery, 2019. [5] o. f. c. rights (ocr), “hipaa compliance and enforcement,” may 2008. library catalog: www.hhs.gov. [6] t. d. breaux, m. w. vail, and a. i. anton, “towards regulatory compliance: extracting rights and obligations to align requirements with regulations,” in 14th ieee international requirements engineering conference (re’06), pp. 49–58, 2006. [7] s. avancha, a. baxi, and d. kotz, “privacy in mobile technology for personal healthcare,” acm comput. surv., vol. 45, dec. 2012. [8] w. shen, c. lin, and a. marcus, “using traceability links to identifying potentially erroneous artifacts during regulatory reviews,” in 2013 7th international workshop on traceability in emerging forms of software engineering (tefse), pp. 19–22, 2013. [9] a. velasco and j. h. aponte melo, “recovering fine grained traceability links between software mandatory constraints and source code,” in applied informatics (h. florez, m. leon, j. m. diaz-nafria, and s. belli, eds.), (cham), pp. 517–532, springer international publishing, 2019. [10] “health insurance portability and accountability act of 1996 (hipaa) | cdc,” feb. 2019. library catalog: www.cdc.gov. [11] t. breaux and a. antón, “analyzing regulatory rules for privacy and security requirements,” ieee transactions on software engineering, vol. 34, no. 1, pp. 5–20, 2008. [12] n. kiyavitskaya, n. zeni, t. d. breaux, a. i. antón, j. r. cordy, l. mich, and j. mylopoulos, “automating the extraction of rights and obligations for regulatory compliance,” in conceptual modeling er 2008 (q. li, s. spaccapietra, e. yu, and a. olivé, eds.), (berlin, heidelberg), pp. 154–168, springer berlin heidelberg, 2008. [13] t. d. breaux and a. i. antón, “a systematic method for acquiring regulatory requirements : a frame-based approach,” 2007. [14] n. zeni, l. mich, j. mylopoulos, and j. r. cordy, “applying gaiust for extracting requirements from legal documents,” in 2013 6th international workshop on requirements engineering and law (relaw), pp. 65–68, 2013. [15] t. alshugran and j. dichter, “extracting and modeling the privacy requirements from hipaa for healthcare applications,” in ieee long island systems, applications and technology (lisat) conference 2014, pp. 1–5, 2014. paradigmplus (2020) 1:2 automated fine grained traceability links recovery between high level requirements and source code implementations 39 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [16] j. c. maxwell and a. i. antón, “checking existing requirements for compliance with law using a production rule model,” in 2009 second international workshop on requirements engineering and law, pp. 1–6, 2009. [17] j. huang, o. gotel, and a. zisman, eds., software and systems traceability. london: springerverlag, 2012. [18] f. fasano, “fine-grained management of software artefacts,” in 2007 ieee international conference on software maintenance, (los alamitos, ca, usa), ieee computer society, oct 2007. [19] a. marcus and j. i. maletic, “recovering documentation-to-source-code traceability links using latent semantic indexing,” in proceedings of the 25th international conference on software engineering, icse ’03, (usa), p. 125–135, ieee computer society, 2003. [20] g. antoniol, g. canfora, g. casazza, a. de lucia, and e. merlo, “tracing object-oriented code into functional requirements,” in proceedings iwpc 2000. 8th international workshop on program comprehension, pp. 79–86, 2000. [21] g. antoniol, g. canfora, a. de lucia, and e. merlo, “recovering code to documentation links in oo systems,” in sixth working conference on reverse engineering (cat. no.pr00303), pp. 136–144, 1999. [22] antoniol, canfora, casazza, and de lucia, “information retrieval models for recovering traceability links between code and documentation,” in proceedings 2000 international conference on software maintenance, pp. 40–49, 2000. [23] f. palomba, m. linares-vásquez, g. bavota, r. oliveto, m. di penta, d. poshyvanyk, and a. de lucia, “user reviews matter! tracking crowdsourced reviews to support evolution of successful apps,” in 2015 ieee international conference on software maintenance and evolution (icsme), pp. 291–300, 2015. [24] a. d. lucia, m. d. penta, r. oliveto, a. panichella, and s. panichella, “improving ir-based traceability recovery using smoothing filters,” in 2011 ieee 19th international conference on program comprehension, pp. 21–30, 2011. [25] d. diaz, g. bavota, a. marcus, r. oliveto, s. takahashi, and a. de lucia, “using code ownership to improve ir-based traceability link recovery,” in 2013 21st international conference on program comprehension (icpc), pp. 123–132, 2013. [26] s. deerwester, s. t. dumais, g. w. furnas, t. k. landauer, and r. harshman, “indexing by latent semantic analysis,” journal of the american society for information science, vol. 41, no. 6, pp. 391–407, 1990. [27] s. t. dumais, “improving the retrieval of information from external sources,” behavior research methods, instruments, & computers, vol. 23, pp. 229–236, june 1991. [28] b. dit, m. revelle, m. gethers, and d. poshyvanyk, “feature location in source code: a taxonomy and survey,” j. softw. evol. process., vol. 25, pp. 53–95, 2013. [29] a. qusef, g. bavota, r. oliveto, a. d. lucia, and d. binkley, “recovering test-to-code traceability using slicing and textual analysis,” journal of systems and software, vol. 88, pp. 147 – 168, 2014. [30] b. dagenais and m. p. robillard, “recovering traceability links between an api and its learning resources,” in 2012 34th international conference on software engineering (icse), pp. 47–57, 2012. [31] b. sharif and j. i. maletic, “using fine-grained differencing to evolve traceability links,” in in gct’07, pp. 76–81, acm, 2007. paradigmplus (2020) 1:2 40 alejandro velasco and jairo aponte . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [32] w. e. wong, s. s. gokhale, j. r. horgan, and k. s. trivedi, “locating program features using execution slices,” in proceedings 1999 ieee symposium on application-specific systems and software engineering and technology. asset’99 (cat. no.pr00122), pp. 194–203, 1999. [33] “itrust | medical free/libre and open source software.” [34] “start [itrust].” [35] “home documentation openmrs wiki.” [36] “oscar emr user’s manual — site.” [37] “tapas home.” [38] m. cohn, user stories applied: for agile software development. boston: addison-wesley professional, edición: 1 ed., mar. 2004. [39] j. garzas, object-oriented design knowledge: principles, heuristics and best practices. hershey, pa: idea group publishing, july 2006. [40] p. azimi and p. daneshvar, “an efficient heuristic algorithm for the traveling salesman problem,” in advanced manufacturing and sustainable logistics (w. dangelmaier, a. blecken, r. delius, and s. klöpfer, eds.), (berlin, heidelberg), pp. 384–395, springer berlin heidelberg, 2010. [41] z. w. geem, j. h. kim, and g. loganathan, “a new heuristic optimization algorithm: harmony search,” simulation, vol. 76, pp. 60–68, feb. 2001. publisher: sage publications ltd stm. [42] h. agrawal, j. r. horgan, s. london, and w. e. wong, “fault localization using execution slices and dataflow tests,” in proceedings of sixth international symposium on software reliability engineering. issre’95, pp. 143–151, 1995. [43] “amazon.com: head first object-oriented analysis and design (0636920008675): brett d. mclaughlin, gary pollice, dave west: books.” [44] s. m. sutton, “aspect-oriented software development and software process,” in unifying the software process spectrum (m. li, b. boehm, and l. j. osterweil, eds.), (berlin, heidelberg), pp. 177–191, springer berlin heidelberg, 2006. [45] b. fluri, m. wursch, and h. c. gall, “do code and comments co-evolve? on the relation between source code and comment changes,” in 14th working conference on reverse engineering (wcre 2007), pp. 70–79, 2007. [46] b. fluri, m. würsch, e. giger, and h. c. gall, “analyzing the co-evolution of comments and source code,” software quality journal, vol. 17, pp. 367–394, dec. 2009. [47] r. c. martin, agile software development, principles, patterns, and practices. upper saddle river, n.j: pearson, edición: 1st ed., oct. 2002. [48] d. binkley, m. davis, d. lawrie, j. i. maletic, c. morrell, and b. sharif, “the impact of identifier style on effort and comprehension,” empirical software engineering, vol. 18, pp. 219–276, apr. 2013. [49] m. ohba and k. gondow, “toward mining "concept keywords" from identifiers in large software projects,” in proceedings of the 2005 international workshop on mining software repositories, msr ’05, (st. louis, missouri), pp. 1–5, association for computing machinery, may 2005. paradigmplus (2020) 1:2 automated fine grained traceability links recovery between high level requirements and source code implementations 41 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [50] y. goldberg and o. levy, “word2vec explained: deriving mikolov et al.’s negative-sampling word-embedding method,” 2014. [51] s. arlt, a. podelski, and m. wehrle, “reducing gui test suites via program slicing,” in proceedings of the 2014 international symposium on software testing and analysis, issta 2014, (new york, ny, usa), p. 270–281, association for computing machinery, 2014. [52] k. b. gallagher and j. r. lyle, “using program slicing in software maintenance,” ieee transactions on software engineering, vol. 17, no. 8, pp. 751–761, 1991. [53] i. mastroeni and d. zanardini, “abstract program slicing: an abstract interpretation-based approach to program slicing,” acm trans. comput. logic, vol. 18, feb. 2017. [54] m. weiser, “program slicing,” in proceedings of the 5th international conference on software engineering, icse ’81, p. 439–449, ieee press, 1981. paradigmplus (2020) 1:2 introduction related work traceability links recovery fine-grained traceability link recovery techniques limitations of adopted strategies dataset description healthcare systems extracting requirements from high level artifacts source code extraction heuristics (h1) heuristic 1: developers tend to use terms present in high-level requirements when writing source code (h2) heuristic 2: comments in source code are a valuable source of information that can be exploited to increase the accuracy of any traceability algorithm (h3) heuristic 3: subjects in texts that describe a high-level software requirement regularly represent entities in the source code (h4) heuristic 4: conventions used by developers to name source code entities can be used to define patterns to improve accuracy in ir algorithms (h5) heuristic 5: delegation is ubiquitous in object-oriented systems and can be used to determine the artifacts that are involved in the implementation of a high-level requirement fine-grained traceability algorithm a1: call graph construction from source code files a2: indexing phase b: information retrieval process c: parsing phase d: program slicing process e: full trace construction phase results discussion conclusions future work