Simulation of intellectual system for evaluation of multilevel test tasks on the basis of fuzzy logic CTE Workshop Proceedings, 2021, Vol. 8: CTE-2020, pp. 507-520 Simulation of intellectual system for evaluation of multilevel test tasks on the basis of fuzzy logic Ivan M. Tsidylo1, Serhiy O. Semerikov2,3,4,5, Tetiana I. Gargula6, Hanna V. Solonetska1, Yaroslav P. Zamora1 and Andrey V. Pikilnyak3 1Ternopil Volodymyr Hnatiuk National Pedagogical University, 2 Maksyma Kryvonosa Str., Ternopil, 46027, Ukraine 2Kryvyi Rih State Pedagogical University, 54 Gagarin Ave., Kryvyi Rih, 50086, Ukraine 3Kryvyi Rih National University, 11 Vitalii Matusevych Str., Kryvyi Rih, 50027, Ukraine 4Institute of Information Technologies and Learning Tools of the NAES of Ukraine, 9 M. Berlynskoho Str., Kyiv, 04060, Ukraine 5University of Educational Management, 52-A Sichovykh Striltsiv Str., Kyiv, 04053, Ukraine 6I. Horbachevsky Ternopil National Medical University, 1 Voli Sq., Ternopil, 46001, Ukraine Abstract. The article describes the stages of modeling an intelligent system for evaluating multilevel test tasks based on fuzzy logic in the MATLAB application package, namely the Fuzzy Logic Toolbox. The analysis of existing approaches to fuzzy assessment of test methods, their advantages and disadvantages is given. The considered methods for assessing students are presented in the general case by two methods: using fuzzy sets and corresponding membership functions; fuzzy estimation method and generalized fuzzy estimation method. In the present work, the Sugeno production model is used as the closest to the natural language. This closeness allows for closer interaction with a subject area expert and build well-understood, easily interpreted inference systems. The structure of a fuzzy system, functions and mechanisms of model building are described. The system is presented in the form of a block diagram of fuzzy logical nodes and consists of four input variables, corresponding to the levels of knowledge assimilation and one initial one. The surface of the response of a fuzzy system reflects the dependence of the final grade on the level of difficulty of the task and the degree of correctness of the task. The structure and functions of the fuzzy system are indicated. The modeled in this way intelligent system for assessing multilevel test tasks based on fuzzy logic makes it possible to take into account the fuzzy characteristics of the test: the level of difficulty of the task, which can be assessed as “easy”, “average", “above average”, “difficult”; the degree of correctness of the task, which can be assessed as “correct”, “partially correct”, “rather correct”, “incorrect”; time allotted for the execution of a test task or test, which can be assessed as “short”, “medium”, “long”, “very long”; the percentage of correctly completed tasks, which can be assessed as “small”, “medium”, “large”, “very large”; the final mark for the test, which can be assessed as “poor”, “satisfactory”, “good”, “excellent”, which are included in the assessment. This approach ensures the maximum consideration of answers to questions of all levels of complexity by formulating a base of inference rules and selection of weighting coefficients when deriving the final estimate. The robustness of the system is achieved by using Gaussian membership functions. The testing of the controller on the test sample brings the functional suitability of the developed model. Keywords: intelligent system, multilevel test tasks, fuzzy test characteristics, fuzzy assessment, Sugeno inference system, 507 CTE Workshop Proceedings, 2021, Vol. 8: CTE-2020, pp. 507-520 1. Introduction Test control is increasingly becoming an integral part of the educational process for all types and levels of educational institutions. Having become widespread in Western European countries and the United States, it is gradually gaining new positions in the domestic higher education. There are many practical implementations of automated testing systems, both in individual disciplines, and universal knowledge assessment systems, fully or partially invariant to specific disciplines and allowing teachers to edit their information content. Analysis of the effectiveness of automated testing in educational institutions shows that the most significant disadvantages of modern approaches to automated testing include [38, p. 4]: • the need to formulate options for answers to test items on the principle of “one is absolutely correct” – “other N are absolutely wrong”; • the primitiveness and inflexibility of the procedures for calculating the final grade, which can be reduced either to determining the ratio of the number of correct answers to the number of questions asked, or to the summation of points assigned for each correct answer; • impossibility of automating various methods of knowledge control, widely used in peda- gogical practice; • significant laboriousness of manual formation of such a set of test tasks and options for answers to each of them, which makes it possible to exclude or minimize the likelihood of presenting the same task to different people while simultaneously checking their knowledge. From this it follows that it is necessary to develop an automated knowledge control system, which requires the use of fundamentally different approaches to the presentation and processing of information based on methods and models developed within the framework of the theory of intelligent computing and knowledge engineering. A lot of studies in pedagogy are devoted to the issue of assessing knowledge. In particular: monitoring the quality of education (Cherednichenko and Yangolenko [7], He and He [16], Igbape and Idogho [19], Leontev et al. [21], Li et al. [22], Muhd Nor et al. [25], Qin et al. [30], Sorour et al. [49], Wei [54], Zhi and Nan [56] and others); development of modern innovative technologies that are included in the knowledge assessment system (Anohina-Naumeca, Straut- mane and Grundspenkis [2], Anohina-Naumeca and Grundspenkis [1], Gierłowski and Nowicki " tsidylo@tnpu.edu.ua (I. M. Tsidylo); semerikov@gmail.com (S. O. Semerikov); gargulavsg@gmail.com (T. I. Gargula); Homenyuk_Hanna@tnpu.edu.ua (H. V. Solonetska); zamora@bigmir.net (Y. P. Zamora); pikilnyak@gmail.com (A. V. Pikilnyak) ~ http://tnpu.edu.ua/faculty/IPF/0002.php (I. M. Tsidylo); https://kdpu.edu.ua/semerikov (S. O. Semerikov); https://osta.tdmu.edu.ua/struktura/gargula-tetana-igorivna (T. I. Gargula); http://tnpu.edu.ua/faculty/fizmat/gomenyuk-ganna-volodimir-vna.php (H. V. Solonetska); http://tnpu.edu.ua/faculty/IPF/zamora-yaroslav-petrovich.php (Y. P. Zamora); http://www.knu.edu.ua/fakultety/fakul-tet-mehanichnoi-inzhenerii-ta-transportu/dekanat (A. V. Pikilnyak) � 0000-0002-0202-348X (I. M. Tsidylo); 0000-0003-0789-0272 (S. O. Semerikov); 0000-0003-3335-0501 (T. I. Gargula); 0000-0002-2527-8653 (H. V. Solonetska); 0000-0001-6470-8233 (Y. P. Zamora); 0000-0003-0898-4756 (A. V. Pikilnyak) CTE Workshop Proceedings © Copyright for this paper by its authors, published by Academy of Cognitive and Natural Sciences (ACNS). This is an Open Access article distributed under the terms of the Creative Commons License Attribution 4.0 International (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. 508 mailto:tsidylo@tnpu.edu.ua mailto:semerikov@gmail.com mailto:gargulavsg@gmail.com mailto:Homenyuk_Hanna@tnpu.edu.ua mailto:zamora@bigmir.net mailto:pikilnyak@gmail.com http://tnpu.edu.ua/faculty/IPF/0002.php https://kdpu.edu.ua/semerikov https://osta.tdmu.edu.ua/struktura/gargula-tetana-igorivna http://tnpu.edu.ua/faculty/fizmat/gomenyuk-ganna-volodimir-vna.php http://tnpu.edu.ua/faculty/IPF/zamora-yaroslav-petrovich.php http://www.knu.edu.ua/fakultety/fakul-tet-mehanichnoi-inzhenerii-ta-transportu/dekanat https://orcid.org/0000-0002-0202-348X https://orcid.org/0000-0003-0789-0272 https://orcid.org/0000-0003-3335-0501 https://orcid.org/0000-0002-2527-8653 https://orcid.org/0000-0001-6470-8233 https://orcid.org/0000-0003-0898-4756 https://acnsci.org/cte https://creativecommons.org/licenses/by/4.0 https://acnsci.org CTE Workshop Proceedings, 2021, Vol. 8: CTE-2020, pp. 507-520 [13], Grundspenkis [14], Schmuck, Sima and Szöllosi [41], Szöllosi, Sima and Schmuck [50] and others); the use of a multi-point scale for assessing knowledge, abilities, and skills (Bespalko [5], Linn [23] and others); theoretical approaches to the assessment of students’ knowledge, their development and improvement (Clotfelter, Ladd and Vigdor [8], Falchikov and Boud [11], Falchikov and Goldfinch [12], Host, Regnell and Wohlin [17], Hwang and Chang [18], Newble and Jaeger [27], Osadchyi et al. [28], Rust, Price and Berry [39], Scouller [42], Topping [53], Wiliam et al. [55] and others); evaluation of test results in an adaptive automated testing system, taking into account the ambiguity of the formulations of answers (Barker [3], Phankokkruad and Woraratpanya [29], Rudinskiy [38] and others). In [6] we substantiated the structural model of the neuro-fuzzy system of professional selection of students for training in IT specialties by studying the psychological characteristics, personal qualities and factual knowledge, skills and abilities of students as a unity of fuzzy and stochastic data base of the intellectual system. The issue of using fuzzy logic to describe the indicators of expert competence assessment using linguistic variables instead of numerical ones or in addition to them and the development of Sugeno’s intelligent system for determining expert competence was covered by us in [52]. The process of modeling intelligent systems based on fuzzy logic in various fields and analysis of the effectiveness of systems implemented in MATLAB is disclosed in the works of: Taylor [51] – fuzzy logic methodology, which is widely used in research and engineering practice and education, Lutsyk et al. [24] – use of parametric identification and adaptive neuro-fuzzy technologies to determine energy efficient modes of production equipment, Shtovba – the theory of fuzzy identification, methods of fuzzy clustering and their application for fuzzy rule extraction, as well as the method of decision-making in fuzzy conditions based on the merger of goals and constraints, author’s package solutions for designing fuzzy classifiers, building hierarchical fuzzy systems, training of fuzzy knowledge bases such as Mamdani, as well as for logical output with fuzzy source data [31–37, 40, 43–48]. 2. Materials and methods Models based on fuzzy logic are more flexible, as they mostly allow taking into account the experience and intuition of a specialist in a particular field. They are more adequate to the simulated reality and make it possible to obtain a solution correlated in accuracy with the initial data [20]. As a rule, the following characteristics are referred to fuzzy test characteristics: 1) the level of difficulty of the task, which can be assessed as “easy”, “average”, “above average”, “difficult”; 2) the degree of correctness of the task, which can be assessed as “correct”, “partially correct”, “rather correct”, “incorrect”; 3) time allotted for the execution of a test task or test, which can be assessed as “short”, “average”, “long”, “very long”; 4) percentage of correctly completed tasks, which can be assessed as “small”, “medium”, “large”, “very large”; 5) final mark for the test, which can be assessed as “bad”, “satisfactory”, “good”, “excellent”. 509 CTE Workshop Proceedings, 2021, Vol. 8: CTE-2020, pp. 507-520 Among the fuzzy models for evaluating test results, adaptive models are interesting. In the work of Rudinskiy [38, p. 49], an adaptive model for evaluating the results of a “fuzzy” test is described. The idea is that the set of reference answers for each test item has a fuzzy grading scale. This fuzzy scale corresponds to the normalized numerical scale (1, 𝑡1, 𝑡2, 𝑡3, 0), where 𝑡𝑖 ∈ (0, 1), 𝑖 = 1..3. All answers, except for the correct one, are assigned a subsequent question with a subset of answers. If an inaccurate answer is given to question 𝐷 at the 𝑖-th step of testing, a clarifying question is asked next, and the subset of answers contains both “more correct” (“correct”, “incomplete”) and “less correct” (“uncertain”, “wrong”) answers. If this question is answered differently from the correct one, no further additional questions are asked (otherwise the laboriousness of compiling such a structure of questions with subsets of answers to them would be very great), testing goes to the next step (question). Thus, the testing process can be represented as a movement along a directed graph, where vertices are questions, and arcs are transitions from the previous question to the next. An adaptive testing model using the apparatus of fuzzy logic is considered by Duplik [10, p. 60]. As a scale for evaluating test results, a 12-point scale proposed by Bespalko [5] is used. At the same time, the author proposes a correspondence between the percent of correct answers of the student and estimates on 12-point and 5-point scales, which, in turn, correspond to fuzzy concepts. Danilova [9, p. 17] developed an adaptive fuzzy model for evaluating the results of automated testing with division of tasks according to the levels of assimilation, proposed by Bespalko [5]. The paper presents models for evaluating test results: formalization of question-answer relations in test tasks according to the levels of assimilation is carried out for recognizing the answers of the tested person and formal presentation of test results; the scaling of the value estimates of the test items was performed; the bases of rules of fuzzy productions for evaluating test items of closed and open types have been developed; in order to ensure the adaptability of testing, a base of rules for fuzzy products has been developed for ranking tasks in the test; the calculation of the integral assessment of the test performance was done based on the assessment results of each test task. The fuzzy inference for evaluating the test results, based on the Mamdani method of fuzzy inference, is described. Belov [4] considers the problem of building an automated testing system (ATS) with the analysis of the respondent’s answers in natural language (NL). To recognize the responses of the person and the reference in the automated testing system, a linguistic analyzer module has been developed, which processes text in NL. The result of the surface-syntactic analysis of the phrases of the reference and user answers are syntactic dependency trees, including the word forms of the phrase, with the definition for each of them morphological descriptors and syntactic properties that combine words into syntactic fragments and groups. A limitation of the presented comparison model is the use of well-formed sentences. A sentence that is not well-formed is discarded by the linguistic analyzer with the requirement to the respondent to reformulate the answer. Each type of response is associated with a so-called syntactic template (SynT), which determines a set of typical syntactic constructions of a sentence and their significance. The obtained result – the degree of correspondence (relevance of phrases) – is taken as the degree of “correctness” of the respondent’s answer on the scale [0; 1]. 510 CTE Workshop Proceedings, 2021, Vol. 8: CTE-2020, pp. 507-520 3. Results Thus, all the methods for evaluating test methods that we have considered have both advantages and disadvantages, which we have summarized in table 1 for clarity. Table 1 Advantages and disadvantages of test assessment methods Author Advantages Disadvantages Rudinskiy [38] The introduction of fuzziness in the orga- nization of the adaptive test, which allows the compilers of the test at the stage of its creation for each test task to build a hierarchical structure of questions in the form of a directed graph. When evaluating test tasks and the test, the apparatus of fuzzy logic is not used, and the obtained linguistic values are sim- ply projected onto a normalized numerical scale. The values obtained on this scale determine the degree of correctness of the answers, which are substituted into a spe- cially designed formula to obtain the final grade. Duplik [10] The use of a fuzzy logic apparatus to ob- tain an integral assessment of test results. The integral assessment is influenced by such indistinct characteristics of the test as the current level of training, the per- centage of correct answers, the complex- ity of the task, and the time it takes to complete the task. The 12-point assessment scale, proposed by V. P. Bespalko, is used only to evenly distribute the traditional 5-point scale on it and is not tied to the levels of assimila- tion of knowledge. Danilova [9] The sophistication of models for assessing test tasks, adaptive testing, integral assess- ment of test results. The set of fuzzy production rules for eval- uating test tasks with an open-ended ques- tion is applicable only to test tasks of the “Substitution” type. Belov [4] Revealed classification of question types and corresponding types of answers in nat- ural language. The graph comparison method is very la- bor intensive and complex. Firstly, the syntactic templates of all reference an- swers must be built in advance, and sec- ondly, the proximity of two phrases is de- termined on the scale [0; 1] by means of a complex algorithm, which would be easier to do using the apparatus of fuzzy logic. The considered methods for assessing students are presented in the general case by two methods: using fuzzy sets and corresponding membership functions; fuzzy estimation method and generalized fuzzy estimation method. The assessment system should be regularly reviewed and improved to ensure its suitability to assess students impartially and fairly. It makes sense to use a fuzzy model to describe an object when we do not have its analyt- ical description, or it is too cumbersome to use, but at the same time there is a sufficiently large amount of experimental data on the behavior of an object and/or heuristic rules for its functioning. 511 CTE Workshop Proceedings, 2021, Vol. 8: CTE-2020, pp. 507-520 In this work, the Sugeno production model is used as the closest to the natural language. This closeness allows for closer interaction with a subject area expert and build well-understood, easily interpreted inference systems. It is important for us to develop an assessment strategy based on fuzzy sets, which requires careful consideration of the factors included in the assessment. These include: the level of difficulty of the task, the degree of correctness of the task, the final mark for the test, which can be assessed as “bad”, “satisfactory”, “good”, and “excellent”. The system is presented in the form of a block diagram of fuzzy logical nodes in figure 1 and consists of four input variables, corresponding to the levels of knowledge assimilation and one initial one. With this method, the system contains two nodes. The first node takes into account the level of complexity of the task and the degree of correctness of the task, depending on the supported task type of the automated system that is used for testing, for example Moodle [26]. The next three nodes behave like a fuzzy logic controller with two inputs with corresponding weights and one output, as in figure 2. 4. Fuzzy system implementation From the subject expert, we get the value of the matrix and the dimensions that describe the degree of importance of each question in the fuzzy domain, that is, the set of all allowed atomic values of the matrix column. The clear values are given as a vector. In the first node, the resulting data will be the experimental data, while the next nodes work as a fuzzy controller, the input of which is the output of the previous node (corresponding to the levels). The output of each node can be in the form of fuzzy values or in the form of linguistic variables. Each node has weighted coefficients that can be set equal to one with the equal influence of each input parameter. The output occurs according to the inference mechanism of the Sugeno fuzzy system. Here is a description of the system. System name: Correctness. Input variables: Level 1, Level 2, Level 3, Level 4. Initial variable: Final grade. The names of the terms of input variables: correct, wrong. The names of the terms of the original variable: correct, almost correct, partly correct, rather correct, probably wrong, wrong, zero. Fuzzy membership functions of the system are defined in the interval [0; 100] (see figure 3), the parameters of the input and initial ones, respectively, are given in tables 2 and 3. Set of rules “ If . . . then”: 1. If (level1 is wrong) and (level2 is wrong) and (level3 is wrong) and (level4 is wrong) then (final grade is zero) (1) 2. If (level1 is wrong) and (level2 is wrong) and (level3 is wrong) and (level4 is correct) then (final grade is probably wrong) (1) 3. If (level1 is wrong) and (level2 is wrong) and (level3 is correct) and (level4 is wrong) then (final grade is probably wrong) (1) 4. If (level1 is wrong) and (level2 is wrong) and (level3 is correct) and (level4 is correct) then (final grade is partly correct) (1) . . . 512 CTE Workshop Proceedings, 2021, Vol. 8: CTE-2020, pp. 507-520 Figure 1: Block diagram of a fuzzy estimation system. 14. If (level1 is correct) and (level2 is correct) and (level3 is wrong) and (level4 is correct) then (final grade is almost correct) (1) 15. If (level1 is correct) and (level2 is correct) and (level3 is correct) and (level4 is wrong) then (final grade is partly correct) (1) 16. If (level1 is correct) and (level2 is correct) and (level3 is correct) and (level4 is correct) then (final-grade is correct) (1) As a result of modeling this system in the MATLAB application package, in particular the Fuzzy Logic Toolbox package, we obtained the response surfaces of the system at constant values of the input variables level3 and level4 equal to 50: in figure 4a – manually configured by an expert; in figure 4b – configured according to the ANFIS algorithm. Analysis of the response surface of a manually tuned system shows incorrect operation at intervals corresponding to 513 CTE Workshop Proceedings, 2021, Vol. 8: CTE-2020, pp. 507-520 Figure 2: Node of presentation in the form of a fuzzy logical controller. Figure 3: Membership functions of input linguistic variables of a fuzzy system. intermediate values of functional membership such as constants of the output variable. To eliminate these differences, the fuzzy system was trained using the ANFIS algorithm based on the training sample. Training program: 514 CTE Workshop Proceedings, 2021, Vol. 8: CTE-2020, pp. 507-520 Table 2 Parameters of membership functions of initial variables Name Type Parameters correct constant 100 almost correct constant 80 partly correct constant 65 rather correct constant 50 probably wrong constant 35 wrong constant 20 zero constant 0 Figure 4: The surface of the system response at constant values of the input variables level3 and level4 equal to 50: a – manually configured by an expert; b – configured according to the ANFIS algorithm. initfis = resdfis (’correctness’); (learn, error) = anfis (tr_data, initfis, 10); where the initial parameters: learn – a tuned system of the Sugeno type, the parameters of which minimize the error on the training set; error – system error at each training iteration; input parameters: tr_data – training sample; initfis – the original fuzzy output system; number 10 is responsible for the number of training iterations. As can be seen from figure 4b, the trained system according to the ANFIS algorithm reproduces the expert’s opinion as accurately as possible, which makes it possible, accordingly, to more accurately formulate the final assessment, taking into account the level of the tasks done correctly. The results of testing the fuzzy system are shown in table 4. 515 CTE Workshop Proceedings, 2021, Vol. 8: CTE-2020, pp. 507-520 Table 3 Fuzzy system testing results Difficulty level 1 Difficulty level 2 Difficulty level 3 Difficulty level 4 Final Grade 20 25 21 36 26 50 65 75 25 52 60 45 25 0 28 98 78 60 10 52 27 35 15 0 18 60 68 50 35 50 85 90 50 54 66 85 87 80 76 81 15 7 2 0 6 56 45 90 84 73 5. Conclusion An intelligent system for assessing multilevel test tasks based on fuzzy logic modeled in this way makes it possible to consider all the above factors using fuzzy logic that are included in the assessment. This approach ensures the maximum consideration of answers to questions of all levels of complexity by formulating a base of inference rules and selection of weighting coefficients when deriving the final grade. The stability of the system is achieved by using Gaussian membership functions, as discussed in [15, p. 14]. We see the prospect of further research in the processing of the information received from the fuzzy system and the formulation of appropriate recommendations for specialists in different fields of knowledge for interpreting the final grade using multilevel test tasks. References [1] Anohina-Naumeca, A. and Grundspenkis, J., 2010. Evaluating students’ concept maps in the concept map based intelligent knowledge assessment system. Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinfor- matics), 5968 LNCS, pp.8–15. Available from: https://doi.org/10.1007/978-3-642-12082-4_2. [2] Anohina-Naumeca, A., Strautmane, M. and Grundspenkis, J., 2010. Development of the scoring mechanism for the concept map based intelligent knowledge assessment system. Sofia, vol. 471, pp.376–381. Available from: https://doi.org/10.1145/1839379.1839446. [3] Barker, T., 2010. An automated feedback system based on adaptive testing: Extending the model. International journal of emerging technologies in learning, 5(2), pp.11–14. Available from: https://doi.org/10.3991/ijet.v5i2.1235. [4] Belov, E.A., 2006. Development of a method and algorithms for testing knowledge based on intelligent processing of answers in natural language. Ph.D. thesis. Bryansk. [5] Bespalko, V., 1960. Requirements of educational films for professional and technical education. Soviet education, 2(3), pp.17–19. Available from: https://doi.org/10.2753/ RES1060-9393020317. 516 https://doi.org/10.1007/978-3-642-12082-4_2 https://doi.org/10.1145/1839379.1839446 https://doi.org/10.3991/ijet.v5i2.1235 https://doi.org/10.2753/RES1060-9393020317 https://doi.org/10.2753/RES1060-9393020317 CTE Workshop Proceedings, 2021, Vol. 8: CTE-2020, pp. 507-520 [6] Buyak, B., Tsidylo, I., Repskyi, V. and Lyalyuk, V., 2018. Stages of conceptualization and formalization in the design of the model of the neuro-fuzzy expert system of professional selection of pupils. Ceur workshop proceedings, 2257, pp.112–121. [7] Cherednichenko, O. and Yangolenko, O., 2013. Towards quality monitoring and evaluation methodology: Higher education case-study. Lecture notes in business information processing, 137, pp.120–127. Available from: https://doi.org/10.1007/978-3-642-38370-0_11. [8] Clotfelter, C., Ladd, H. and Vigdor, J., 2006. Teacher-student matching and the assessment of teacher effectiveness. Journal of human resources, 41(4), pp.778–820. Available from: https://doi.org/10.3368/jhr.xli.4.778. [9] Danilova, S.D., 2008. Evaluation of test results in an adaptive automated testing system. Bulletin of vsstu, (1), pp.12–20. [10] Duplik, S.V., 2004. Model of adaptive testing on fuzzy mathematics. Computer science and education, (11), pp.57–65. [11] Falchikov, N. and Boud, D., 1989. Student self-assessment in higher education: A meta- analysis. Review of educational research, 59(4), pp.395–430. Available from: https://doi.org/ 10.3102/00346543059004395. [12] Falchikov, N. and Goldfinch, J., 2000. Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of educational research, 70(3), pp.287–322. Available from: https://doi.org/10.3102/00346543070003287. [13] Gierłowski, K. and Nowicki, K., 2008. A novel architecture for e-learning knowledge assessment systems. Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), 4823 LNCS, pp.276–287. Available from: https://doi.org/10.1007/978-3-540-78139-4_25. [14] Grundspenkis, J., 2019. Intelligent knowledge assessment systems: Myth or reality. Frontiers in artificial intelligence and applications, 315, pp.31–46. Available from: https://doi.org/10. 3233/978-1-61499-941-6-31. [15] Hameed, I.A. and Sorensen, C.G., 2010. Fuzzy systems in education: A more reliable system for student evaluation. In: A.T. Azar, ed. Fuzzy systems. IntechOpen, pp.1–17. Available from: https://doi.org/10.5772/7216. [16] He, L. and He, W., 2009. Study on the construction of internal monitoring system of chinese independent colleges’ education quality. Genova, pp.191–194. [17] Host, M., Regnell, B. and Wohlin, C., 2000. Using students as subjects - a comparative study of students and professionals in lead-time impact assessment. Empirical software engineering, 5(3), pp.201–214. Available from: https://doi.org/10.1023/A:1026586415054. [18] Hwang, G.J. and Chang, H.F., 2011. A formative assessment-based mobile learning approach to improving the learning attitudes and achievements of students. Computers and education, 56(4), pp.1023–1031. Available from: https://doi.org/10.1016/j.compedu.2010.12.002. [19] Igbape, E. and Idogho, P., 2019. N-dimension data visualization spaces for academic programmes quality monitoring in nigeria higher education. Newswood Limited, vol. 2019-October, pp.238–242. [20] Khvostina, I., Oliinyk, V., Semerikov, S., Solovieva, V., Yatsenko, V. and Kohut-Ferens, O., 2021. Hazards and risks in assessing the impact of oil and gas companies on the environment. Iop conference series: Earth and environmental science, 628(1), p.012027. Available from: https://doi.org/10.1088/1755-1315/628/1/012027. 517 https://doi.org/10.1007/978-3-642-38370-0_11 https://doi.org/10.3368/jhr.xli.4.778 https://doi.org/10.3102/00346543059004395 https://doi.org/10.3102/00346543059004395 https://doi.org/10.3102/00346543070003287 https://doi.org/10.1007/978-3-540-78139-4_25 https://doi.org/10.3233/978-1-61499-941-6-31 https://doi.org/10.3233/978-1-61499-941-6-31 https://doi.org/10.5772/7216 https://doi.org/10.1023/A:1026586415054 https://doi.org/10.1016/j.compedu.2010.12.002 https://doi.org/10.1088/1755-1315/628/1/012027 CTE Workshop Proceedings, 2021, Vol. 8: CTE-2020, pp. 507-520 [21] Leontev, M., Bondarenko, N., Shebzuhova, T., Butko, S. and Egorova, L., 2018. Improving the efficiency of university management: Teacher’s performance monitoring as a tool to promote the quality of education. European research studies journal, 21(2), pp.527–540. Available from: https://doi.org/10.35808/ersj/1020. [22] Li, Y., Li, P., Zhu, F. and Wang, R., 2017. Design of higher education quality monitoring and evaluation platform based on big data. Institute of Electrical and Electronics Engineers Inc., pp.337–342. Available from: https://doi.org/10.1109/ICCSE.2017.8085513. [23] Linn, R., 1998. Validating inferences from national assessment of educational progress achievement-level reporting. Applied measurement in education, 11(1), pp.23–47. Available from: https://doi.org/10.1207/s15324818ame1101_2. [24] Lutsyk, I., Franko, Y., Rak, V., Lutsyk, I., Leshchii, R. and Potapchuk, O., 2019. Mathematical modeling of energy-efficient active ventilation modes of granary. 2019 9th international conference on advanced computer information technologies (acit). pp.105–108. Available from: https://doi.org/10.1109/ACITT.2019.8780109. [25] Muhd Nor, N., Azlan, M., Kiong, S., Mohamad, F., Ismail, A., Kasmin, A., Ahmad, M. and Yokoyama, S., 2014. Development of course management and monitoring system as a quality tools in engineering education. Applied mechanics and materials, 465-466, pp.395–400. Available from: https://doi.org/10.4028/www.scientific.net/AMM.465-466.395. [26] Nechypurenko, P. and Semerikov, S., 2017. VlabEmbed - the new plugin Moodle for the chemistry education. Ceur workshop proceedings, 1844, pp.319–326. [27] Newble, D. and Jaeger, K., 1983. The effect of assessments and examinations on the learning of medical students. Medical education, 17(3), pp.165–171. Available from: https: //doi.org/10.1111/j.1365-2923.1983.tb00657.x. [28] Osadchyi, V.V., Osadcha, K.P., Varina, H.B., Shevchenko, S.V. and Bulakh, I.S., 2021. Specific features of the use of augmented reality technologies in the process of the development of cognitive component of future professionals’ mental capacity. Journal of physics: Conference series. [29] Phankokkruad, M. and Woraratpanya, K., 2008. An automated decision system for com- puter adaptive testing using genetic algorithms. Phuket, pp.655–660. Available from: https://doi.org/10.1109/SNPD.2008.118. [30] Qin, F., Zeng, W., Li, L. and Zhao, R., 2020. Construction of big data monitoring platform for teaching quality under intelligent education. Institute of Electrical and Electronics Engineers Inc., pp.1594–1597. Available from: https://doi.org/10.1109/IWCMC48107.2020. 9148224. [31] Rotshtein, A. and Shtovba, S., 1998. Predicting the reliability of algorithmic processes with fuzzy input data. Cybernetics and systems analysis, 34(4), pp.545–552. Available from: https://doi.org/10.1007/BF02666999. [32] Rotshtein, A. and Shtovba, S., 2001. Fuzzy multicriteria analysis of variants with the use of paired comparisons. Journal of computer and systems sciences international, 40(3), pp.499–503. [33] Rotshtein, A. and Shtovba, S., 2009. Modeling of the human operator reliability with the aid of the Sugeno fuzzy knowledge base. Automation and remote control, 70(1), pp.163–169. Available from: https://doi.org/10.1134/S0005117909010123. [34] Rotshtein, A., Shtovba, S. and Mostav, I., 2001. Fuzzy rule based innovation projects 518 https://doi.org/10.35808/ersj/1020 https://doi.org/10.1109/ICCSE.2017.8085513 https://doi.org/10.1207/s15324818ame1101_2 https://doi.org/10.1109/ACITT.2019.8780109 https://doi.org/10.4028/www.scientific.net/AMM.465-466.395 https://doi.org/10.1111/j.1365-2923.1983.tb00657.x https://doi.org/10.1111/j.1365-2923.1983.tb00657.x https://doi.org/10.1109/SNPD.2008.118 https://doi.org/10.1109/IWCMC48107.2020.9148224 https://doi.org/10.1109/IWCMC48107.2020.9148224 https://doi.org/10.1007/BF02666999 https://doi.org/10.1134/S0005117909010123 CTE Workshop Proceedings, 2021, Vol. 8: CTE-2020, pp. 507-520 estimation. Vancouver, BC, vol. 1, pp.122–126. [35] Rotshteina, A. and Shtovbab, S., 2006. Identification of a nonlinear dependence by a fuzzy knowledgebase in the case of a fuzzy training set. Cybernetics and systems analysis, 42(2), pp.176–182. Available from: https://doi.org/10.1007/s10559-006-0051-1. [36] Rotshtejn, A. and Shtovba, S., 2001. Fuzzy rule based control of a dynamic system. Av- tomatika i vychislitel’naya tekhnika, (2), pp.23–31. [37] Rotshtejn, A. and Shtovba, S., 2002. Influence of methods of defuzzification on speed of tuning the fuzzy model. Kibernetika i sistemnyj analiz, (5), pp.169–176. [38] Rudinskiy, I.D., 2007. Fuzzy knowledge evaluation model as a methodological basis for automation of pedagogical testing. Ieee transactions on education, 50(1), pp.68–73. Available from: https://doi.org/10.1109/TE.2006.888904. [39] Rust, C., Price, M. and Berry, O., 2003. Improving students’ learning by developing their understanding of assessment criteria and processes. Assessment and evaluation in higher education, 28(2), pp.147–164. Available from: https://doi.org/10.1080/02602930301671. [40] Rótshtein, A. and Stovba, S., 2001. Managing a dynamic system by means of a fuzzy knowledge base. Automatic control and computer sciences, 35(2), pp.16–22. [41] Schmuck, B., Sima, D. and Szöllosi, S., 2006. The design space of the implementation of knowledge assessment systems. Sydney, NSW, pp.587–593. Available from: https: //doi.org/10.1109/ITHET.2006.339672. [42] Scouller, K., 1998. The influence of assessment method on students’ learning approaches: Multiple choice question examination versus assignment essay. Higher education, 35(4), pp.453–472. Available from: https://doi.org/10.1023/A:1003196224280. [43] Shtovba, S., 2006. Fuzzy identification on the basis of regression models of parametric membership function. Journal of automation and information sciences, 38(11), pp.36–44. Available from: https://doi.org/10.1615/JAutomatInfScien.v38.i11.40. [44] Shtovba, S., 2007. Ensuring accuracy and transparency of Mamdani fuzzy model in learning by experimental data. Journal of automation and information sciences, 39(8), pp.39–52. Available from: https://doi.org/10.1615/JAutomatInfScien.v39.i8.50. [45] Shtovba, S., 2007. Fuzzy model tuning based on a training set with fuzzy model output values. Cybernetics and systems analysis, 43(3), pp.334–340. Available from: https://doi. org/10.1007/s10559-007-0054-6. [46] Shtovba, S., Pankevich, O. and Nagorna, A., 2015. Analyzing the criteria for fuzzy classifier learning. Automatic control and computer sciences, 49(3), pp.123–132. Available from: https://doi.org/10.3103/S0146411615030098. [47] Shtovba, S. and Pankevych, O., 2018. Fuzzy technology-based cause detection of structural cracks of stone buildings. CEUR-WS, vol. 2105, pp.209–218. [48] Shtovba, S. and Shtovba, E., 2006. Prediction of competitive position of brand product by fuzzy knowledge base. Journal of automation and information sciences, 38(8), pp.69–80. Available from: https://doi.org/10.1615/JAutomatInfScien.v38.i8.70. [49] Sorour, A., Atkins, A., Stanier, C. and Alharbi, F., 2020. Comparative frameworks for monitoring quality assurance in higher education institutions using business intelligence. Institute of Electrical and Electronics Engineers Inc. Available from: https://doi.org/10. 1109/ICCIT-144147971.2020.9213808. [50] Szöllosi, S., Sima, D. and Schmuck, B., 2006. The design space of the services of knowledge 519 https://doi.org/10.1007/s10559-006-0051-1 https://doi.org/10.1109/TE.2006.888904 https://doi.org/10.1080/02602930301671 https://doi.org/10.1109/ITHET.2006.339672 https://doi.org/10.1109/ITHET.2006.339672 https://doi.org/10.1023/A:1003196224280 https://doi.org/10.1615/J Automat Inf Scien.v38.i11.40 https://doi.org/10.1615/JAutomatInfScien.v39.i8.50 https://doi.org/10.1007/s10559-007-0054-6 https://doi.org/10.1007/s10559-007-0054-6 https://doi.org/10.3103/S0146411615030098 https://doi.org/10.1615/J Automat Inf Scien.v38.i8.70 https://doi.org/10.1109/ICCIT-144147971.2020.9213808 https://doi.org/10.1109/ICCIT-144147971.2020.9213808 CTE Workshop Proceedings, 2021, Vol. 8: CTE-2020, pp. 507-520 assessment systems. Sydney, NSW, pp.571–578. Available from: https://doi.org/10.1109/ ITHET.2006.339816. [51] Taylor, A., 2017. Fuzzy logic with matlab: Analyzing, designing, and simulating systems. CreateSpace Independent Publishing Platform. [52] Tereshchuk, H.V. and Tsidylo, I.M., 2018. Automated system of fuzzy identification of expert’s competence for assessing the quality of pedagogical phenomena and processes. Information technologies and learning tools, 64(2), pp.234–244. Available from: https: //doi.org/10.33407/itlt.v64i2.2079. [53] Topping, K., 1998. Peer assessment between students in colleges and universities. Re- view of educational research, 68(3), pp.249–276. Available from: https://doi.org/10.3102/ 00346543068003249. [54] Wei, C., 2013. Higher vocational education quality monitoring system. Lecture notes in electrical engineering, 217 LNEE(VOL. 2), pp.55–60. Available from: https://doi.org/10.1007/ 978-1-4471-4850-0_8. [55] Wiliam, D., Lee, C., Harrison, C. and Black, P., 2004. Teachers developing assessment for learning: Impact on student achievement. Assessment in education: Principles, policy and practice, 11(1), pp.49–65. Available from: https://doi.org/10.1080/0969594042000208994. [56] Zhi, Z. and Nan, Z., 2011. Study on the construction of teaching quality monitoring system for the undergraduate physical education majors. Jilin, pp.758–761. Available from: https://doi.org/10.1109/HHBE.2011.6028937. 520 https://doi.org/10.1109/ITHET.2006.339816 https://doi.org/10.1109/ITHET.2006.339816 https://doi.org/10.33407/itlt.v64i2.2079 https://doi.org/10.33407/itlt.v64i2.2079 https://doi.org/10.3102/00346543068003249 https://doi.org/10.3102/00346543068003249 https://doi.org/10.1007/978-1-4471-4850-0_8 https://doi.org/10.1007/978-1-4471-4850-0_8 https://doi.org/10.1080/0969594042000208994 https://doi.org/10.1109/HHBE.2011.6028937 1 Introduction 2 Materials and methods 3 Results 4 Fuzzy system implementation 5 Conclusion