Available online at www.HighTechJournal.org HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 572 ISSN: 2723-9535 Learner Assessment System in e-Learning with OBE Approach: Activity Performance, Ability Level and Recommendation Wenty D. Yuniarti 1, 2 , Sri Hartati 1* , Sigit Priyanta 1 , Herman D. Surjono 3 1 Department of Computer Science and Electronics, Universitas Gadjah Mada, Yogyakarta, 55281, Indonesia. 2 Department of Information Technology, Universitas Islam Negeri Walisongo, Semarang, 50185, Indonesia. 3 Department of Informatics Engineering Education, Faculty of Engineering, Univeristas Negeri Yogyakarta, Yogyakarta, 55281, Indonesia. Received 13 April 2024; Revised 07 August 2024; Accepted 16 August 2024; Published 01 September 2024 Abstract E-learning can lead learners to achieve learning outcomes if it is designed based on several principles. One is applying assessments that motivate and inform ability levels. In Outcome-based Education (OBE), assessment is integral to the system. However, e-learning has limitations in providing assessment instruments according to needs, such as assessing complex and detailed aspects and accommodating a variety of numerical and linguistic assessment data. Moreover, the presence and involvement of learners affect their performance and learning outcomes. This study proposes a learner assessment system in e-learning with the OBE approach, including learning design, activity performance analysis, ability level determination, and recommendations. This system adds the e-rubric to e-learning to overcome instrument limitations and accommodate comprehensive assessments. Various numerical and linguistic assessment data are unified using 2-tuple fuzzy linguistics, producing ability levels as two tuples. Performance analysis was based on event log data using descriptive statistical technique and alignment-based conformance checking, from frequency, time, and sequence of activity objects, resulting in five activity performance variables. The performance value of each variable is converted into High, Medium, or Low levels. The ability and performance levels are processed using rule-based methods to produce recommendations for learning stages and activity performance directions. The results of this research can be used as input for academic stakeholders and online learning providers and potentially be applied to the advancement of e-learning in higher education. Keywords: Assessment on e-Learning; OBE; 2-Tuple Fuzzy Linguistic; Activity Performance Analysis; Rule-Based. 1. Introduction The spread of technology in learning has occurred massively, marked by the rapid shift toward online learning [1- 3], with e-learning as one of its forms. It is attractive as well as challenging because online learning has consequences for a spatial and temporal gap [4], low teacher attendance rates [5], an autonomous nature with a less robust framework in encouraging learners to learn [6], rising concerns about engagement learners [7], and guaranteeing the attainment of learning outcomes [2, 4]. On the other side, educational institutions are obliged to ensure quality online learning that meets accreditation standards [8, 9]. According to Kemendikbud (2020) [2], using online learning in an appropriate, systematic, logical, and structured manner based on several principles can guide learners to achieve learning outcomes. One of the principles is to apply assessment that motivates and informs future practical guidance [2]. Assessment is an indispensable component in e-learning, and this is in line with the presence of outcome-based * Corresponding author: shartati@ugm.ac.id http://dx.doi.org/10.28991/HIJ-2024-05-03-03 ➢ This is an open access article under the CC-BY license (https://creativecommons.org/licenses/by/4.0/). © Authors retain all copyrights. https://creativecommons.org/licenses/by/4.0/ https://orcid.org/0000-0002-5633-0709 https://orcid.org/0000-0002-5511-3921 https://orcid.org/0000-0002-1673-8582 https://orcid.org/0000-0002-2720-2206 HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 573 Education (OBE) as a design methodology and the current global learning curriculum [2, 10], where OBE makes assessment an integral part of learning by prioritizing alignment between learning outcomes, process, and assessment [10]. Unfortunately, assessment problems remain in e-learning, such as those that are only product-oriented, not process, not yet comprehensive, and do not provide feedback [6, 11–13]. According to Lara et al. (2019) [14], various techniques, approaches, or frameworks exist to conduct assessments or in part of the learner assessment process in e-learning, such as e-generic assessment, blockchain, gamification- based, IoT, fuzzy logic, data mining, and process mining. Lara et al. (2019) [14] and Jacob & Henriques [15] recommend data mining techniques to dig up relatively simple patterns of assessment variables in databases, such as profiles, quiz scores, attendance, and access frequency. Unfortunately, these studies are still limited to predictive assessment. The different characteristics of the e-learning environment vs. face-to-face raise the fact that the behavior of the presence and involvement of learners is a potential problem because both impact performance and learning outcomes [16, 17]. Over the decades, the activity or process data, called event logs, have become readily available [4, 16, 18-21]. Event data has three main attributes, namely Case ID, activity, and timestamps, where all three can be explored and knowledge taken from various perspectives [1]. Many performance-related things can be revealed, such as procrastination, involvement, retention, etc. Unfortunately, using event logs for this assessment is still limited to predictive assessment [22–25]. In contrast to data mining, [22, 26] introduced process mining, where event logs can be used to discover, monitor, and improve processes [27]. In realizing process-oriented and thorough assessments, e-learning platforms have limitations in providing specific tools that allow teachers to assess learners according to their needs [16]. Meanwhile, in OBE, collecting information to measure learning outcomes requires comprehensively integrating various techniques and instruments. In addition to analyzing process activities, realizing a comprehensive assessment is also essential to e-learning. The OBE approach responds to the need for process and thorough assessment by recommending using rubric instruments to assess learners [2]. A rubric in a more efficient and sophisticated program, such as an online system called an Electronic Rubric or e-rubric [28]. A rubric is widely used to assess numerical data [29, 30]. According to Brookhart (2018) [31] and Ho et al. (2020) [32], rubrics should be written in descriptive or linguistic language so that learners can imagine their performance level and know what the achievements should be. However, the rubric’s ability to accommodate assessments in linguistic data engenders problems related to how combining this linguistic data with other assessment data in numerical form. According to Herrera & Martinez (2000) [33] and Herrera & Martinez (1996) [34], proper data processing is necessary to avert losing significant information in the assessment. Meta- analysis research mentions several studies using fuzzy logic to process assessment data [35]. Some research uses fuzzy logic to process a combination of numerical assessment data [8, 9, 36]. Andayani (2017) [11] processes numerical and linguistic data using 2-tuple fuzzy linguistics. Unfortunately, this numerical and linguistic data combination is implemented in face-to-face learning (F2F), not e-learning. Recommendations are an essential part that accompanies the assessment while simultaneously describing the learning cycle as a continuous process [32]. Several studies build e-learning and recommend materials and learning paths [6, 12, 13, 37]. Due to the importance of the learning process in OBE construction, recommendations that can guide students to achieve learning outcomes are necessary. This study proposes an e-learner assessment system that can answer the following needs: (1) how to formulate a learning design with the OBE approach so that outcomes, processes, and assessment are aligned; (2) how to realize a comprehensive assessment through analysis of activity performance by utilizing event logs data; (3) how to realize a comprehensive assessment by overcoming the assessment instrument limitations in e-learning through an e-rubric accompanied by a mechanism for unifying numerical and linguistic assessment data for presenting ability level; and 4) how to process the results of activity performance analysis and ability level into recommendations. The principal contribution of this paper is to provide a learner assessment system in e-learning designed with the OBE approach, capable of presenting comprehensive assessment results based on analysis of activity performance and ability level, and able to provide recommendations to guide the attainment of learning outcomes. 2. Literature Review 2.1. Learner Assessment in e-Learning with OBE Approach As a learning environment, e-learning has two characteristics. First, based on technology. E-learning is a web-based system that delivers and manages learning, called the Learning Management System (LMS) [5]. The LMS provides assessment features, like quizzes, assignments, surveys, etc. In essence, assessment is all the ways to assess the performance of individuals or groups. On computer-based platforms, such as LMS, all user engagements that produce data used in the assessment process are called assessment items [38]. LMS collects and stores activity process data or event logs. Event logs are assessment items and datasets that can be analyzed to support assessment decisions [14, 16, 19, 20, 21]. The second characteristic is that e-Learning has a pedagogical-based methodological design [5], and OBE references current pedagogical methodological designs. Previous studies designed e-learning activities with OBE but HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 574 still needed to align them with assessments according to OBE needs [39–41]. OBE recommends a rubric instrument in a comprehensive process assessment framework. The rubric is appropriate for assessing complex skills because rubrics can be constructed based on aspects or dimensions. Each dimension can be scored numerically or descriptively in linguistics, so it shows a role in detecting skill strength and weakness [31, 32]. Rubrics that are implemented electronically are called e-rubrics [29]. Other researchers use e-rubrics for numerical skills assessment. Utilizing e- rubrics in linguistic data and unifying them with other assessment data are challenges in the comprehensive assessment framework [28, 30]. 2.2. Activity Performance Analysis Utilizing data mining techniques to predict e-learners performance based on various combinations of relevant attributes has been carried out by several studies, such as [25] using the C4.5 and Naïve Bayes algorithms based on profiles and data access or clickstream, [23] using the Neural Network algorithm based on attendance, quiz scores, frequency of viewing classes and materials, [24] using Back Propagation Neural Network based on performance and non-performance attributes, and [42] using Fuzzy Association Rule Mining based on previous academic record, attendance, mid and end marks. According to Cerezo et al. (2017) [16], conditions in e-learning, such as engagement, procrastination, retention, and the risk of failure to complete assignments, impact performance, and learning outcomes. Data event logs help minimize this situation. Event log attributes can be managed into relevant objects added as predictors in predictive models. Several studies have added the management of timestamps attribute to detect the risk of quitting and failing [18] and evaluating procrastination behavior. Other research adds clickstream to the activity attribute for early prediction of withdrawal [19], detect failure [20], or identify low engagement [21]. Over the past ten years, process mining techniques have been widely used as performance analysis techniques [43]. Data mining aims to uncover relatively simple patterns within extensive datasets; it is different from process mining, which describes end-to-end processes. Varying dimensions, such as time, cost, or quality, can determine the performance of a process. In process mining, performance analysis can be based on a single object, such as frequency or time, and a combination of single objects, using several modeling techniques, such as business strategy models, Petri Net, directly following models, and alignment [44]. Several studies have used event logs with conformance-checking techniques to see performance through activity conformance detection, such as [22] in learner learning activities, [26] in service processes in health facilities, and [45] in the dwell time of the loading and unloading process. Some researchers propose alignment-based conformance checking to detect conformity of activities to replace token-based [27, 45, 46]. Alignment-based conformance checking can detect the conformity of learning activities by replaying the trace event model of the activity design process with the learner’s event logs. 2.3. Ability Level and Recommendation The assessment ends with the conclusion of learning outcomes in the level of attainment as a report to learners or stakeholders in need. The level of learning attainment shows a person’s ability in specific skills [2]. Concerning teaching and learning situations, [47] categorizes the three ability levels: High, Medium, and Low. This ability level categorization can be done empirically or hypothetically according to the purpose and condition of the data. Assessment reports result from data processing. This reporting requires appropriate techniques to avoid losing important information in the assessment [33, 34]. Meta-analysis research [35] states that several studies utilize fuzzy logic to process numerical assessment data or a combination of numerical data [8, 9, 11, 26, 36]. Andayani (2017) [11] proposes unifying numerical and linguistic data using a computational linguistic model, 2-tuple fuzzy linguistic. Unfortunately, this research is implemented in face-to-face learning rather than in e-learning. Assessments and recommendations are interrelated. Recommendations are part of a continuous learning cycle, giving direction and strengthening, aiming for higher-quality learning. Agustianto et al. (2016) [12] proposed adaptive learning with learning path recommendations based on metacognitive assessment; [6] suggests e-learning with material recommendations based on learning styles and ability levels; [48] provides learning module recommendations based on level of knowledge with content-based filtering methods; and [49] presents recommendations based on automated assessments. In the OBE approach, reporting on learning outcomes accompanied by recommendations is very necessary because it will help learners guide the attainment of learning outcomes. 3. Proposed Method This section discusses the methodology of this study. Figure 1 presents the learner assessment system in e-learning with the OBE approach [50]. The system consists of 4 parts, namely learning design accompanied by the provision of e-rubric instruments (A), analysis of activity performance (B), determination of ability levels (C), and provision of recommendation (D). The subsequent sections will cover each of the stages in detail. HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 575 3.1. Learning Design with OBE Approach and Assessment Variables In this study, 140 Information Technology undergraduate learners participated in the Basic Programming course through LMS e-learning. This course has CLOs (Course Learning Outcomes), LLOs (Lesson Learning Outcomes), and LLO indicators. All three are interrelated, as shown in part A of Figure 1 [50]. This course has 4 CLOs and 6 LLOs. The relation among CLO, LLO, and LLO indicators establishes the basis for learning design, which is structured in the following stage: Figure 1. Learner assessment system in e-learning with an OBE approach. • Formulate CLOs to achieve learners’ abilities as presented in Table 1. CLOs include elements of attitude, knowledge, and skills [2]. • Developing LLOs using CLOs. LLOs delineates the stages of learning, demonstrate the final ability at each stage, and contribute cumulatively to CLOs. HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 576 • Identify indicators of achievement of LLOs. Table 2 provides an example of 3 LLOs out of the total LLOs in this course, namely LLO 1, LLO 2, and LLO 3. Based on the description of CLOs in Table 1 and the LLOs and indicators in Table 2, the learning design for this course is formulated, as presented in Table 3. The learning design provides details for each LLO, including the associated CLO, time duration, sequence of learning activities, assessment technique, and instruments. The learning designs presented are limited to examples of LLO 1, LLO 2, and LLO 3. Table 1. Course learning outcomes CLO Description of learning outcomes Element 1 Demonstrate an attitude of discipline as a form of lifelong learning according to the area of expertise Attitude 2 Mastering the concepts and theories of information technology Knowledge 3 Able to apply logical and systematic thinking in problem-solving or making decisions in the field of expertise Skills 4 Mastering programming concepts and methods as the foundation for data processing in information technology applications Skills LLO is implemented in several sequential learning activities to guide learners to attain learning outcomes. The sequence of learning activities is 1) viewing video material, 2) viewing PDF document material, 3) discussing in forums, 4) doing quizzes, and 5) doing assignments. Based on this sequence, learners are guided to LLOs by viewing video and PDF document material, discussing, and ending with tests and assignments. Table 2. LLO and LLO indicators LLO LLO description LLO indicator 1 Understanding the problem-solving- oriented algorithmic thinking paradigm 1.1 Truth in understanding the concept of algorithmic thinking with algorithms, pseudocode, and the rules for their use 1.2 The accuracy of using algorithms, pseudocode, and flowcharts in solving real-world problems skillfully 2 Understand general concepts and basic elements of programming languages 2.1 Truth in understanding the concept of elements of a programming language according to applicable rules 2.2 The accuracy of using these elements in writing programs in a programming language environment 3 Understand the concepts of branching and looping 3.1 Accuracy in understanding and using a variety of branching and looping constructions well 3.2 The precision in selecting and implementing branching and looping constructs to address practical issues The assessment is carried out by integrating various techniques, namely tests, performance assessments, and observation of disciplinary attitudes. Each technique uses different instruments for data collection, using quizzes, e- rubrics, and event logs. This research proposes adding an e-rubric assessment instrument to take the constraints of e- learning into account in providing specific tools for assessment needs [16], as shown in Figure 2. Table 3. Learning design Week CLO LLO LLO indicators Learning Activity Assessment technique Assessment instrument 1,2 CLO 1 CLO 2 CLO 3 LLO 1 Indicator 1.1 Indicator 1.2 1. Watching video material 2. Watching pdf material 3. Participate in forum 4. Do test 5. Do assignment 1. Written test 2. Performance assessment 3. Observation 1. Quiz 2. e-Rubric 3. Logs recording 3,4 CLO 1 CLO 2 CLO 4 LLO 2 Indicator 2.1 Indicator 2.2 1. Watching video material 2. Watching pdf material 3. Participate in forum 4. Do test 5. Do assignment 1. Written test 2. Performance assessment 3. Observation 1. Quiz 2. e-Rubric 3. Logs recording 5,6,7 CLO 1 CLO 3 CLO 4 LLO 3 Indicator 3.1 Indicator 3.2 1. Watching video material 2. Watching pdf material 3. Participate in forum 4. Do test 5. Do assignment 1. Written test 2. Performance assessment 3. Observation 1. Quiz 2. e-Rubric 3. Logs recording HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 577 Figure 2 is an example of an e-rubric used as an assessment instrument in LLO 2 indicator 2.2 [51]. This e-rubric is used to assess skills in the accuracy of using basic elements in writing programs in a programming language environment, as presented in the LLO description in Table 2. There are four dimensions used to assess the performance, namely a) able to construct correct program flow, b) able to use and write program elements, c) able to apply composition technique in program structure and syntax, and d) the program works and meets all specifications. Each ability dimension is assessed in linguistics, as presented in Figure 2. The learner’s interaction with the LMS is documented in the LMS database logs during the course’s implementation. The learning design guides the activities of the learning process and is recorded naturally in the LMS e-learning. These data records are grouped into two datasets: related to ability level and related to activity performance. • The dataset related to ability level represents the assessment results of the three instruments. First, assessment data from quizzes is used to measure the achievement of knowledge elements. Second, assessment data from e-rubric are used to measure the achievement of the skills element, and third, assessment data from recorded logs are used to measure the achievement of the disciplinary attitude element. Figure 2. E-rubric with several dimensions for assessment instruments. • The dataset related to activity performance is obtained from event logs. Event log analysis requires appropriate data conditions. Labeling learning activities is a way to prepare event log data for further analysis. The sequence of activities in Table 3 is represented by labels, namely label (a) for login activity, (b) for viewed video material, (c) for viewed PDF document, (d) for visit forum, (e) for post to forum, (f) for viewed quiz, (g) for submitted quiz, (h) for viewed assignment, (i) for submitted assignment, and (j) for logout, as presented in Table 4. Table 4 also gives teacher’s note instructions to learners regarding how the activity is carried out. Based on previous studies [8, 9, 18, 19, 20, 27] from several learner activities in Table 4, five variables are determined as relevant activity performance variables, namely frequency of attendance, duration of attendance, frequency of access to video material and PDF documents, the number of posts in the forum, and the conformance of the actual learner’s activities with the design activities that serve as a guide. The presentation of both the description and data collection methods for each variable are in Table 5. Table 4. Learning activities in each LLO Label Activity Instruction notes to learners a Login Login of the course b Viewed video material The activity of viewing video material c Viewed pdf document The activity of viewing pdf document material d Visit the forum Visit forums e Post to forum Post opinions in the forum f Viewed quiz View quiz information g Submitted quiz Submit quizzes h Viewed assignment View task information i Submitted assignment Submit assignments j Logout Logout of the course HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 578 Table 5. Activity performance variables Variables Descriptions Data collecting methods Frequency of attendance Total login-logout (session) in LMS Adding up the number of individual learner’s login time into the e-learning Duration of attendance Time spent in the e-learning LMS Calculating the total amount of time spent between login and logout Frequency of access material Number access course material Adding up the numbers of course material accessed Number of posts to forum Number of online forum post Counting the number of posts a learner has contributed to the discussion forum Activity conformance Deviation between event activity and design activity Conformance between event activity learner and designed activity as a model 3.2. Activity Performance Analysis Event logs from LMS e-learning are used as datasets in the activity performance analysis. Table 6 presents examples of cases, traces, and events from LMS event logs. Event logs have several cases; a trace of events represents each case, and every event is linked to an activity performed for a particular case [27]. For instance, Table 6 presents two cases, namely Case ID 466 and 501. Each case has a different trace. Case ID 466 is a trace with ten events, while Case ID 501 has three events. The case represents several events from login to logout. An event represents a learning activity as presented in Table 6. Table 6. Event logs from LMS Case id Event id Activity Timestamp 466 45900107 \user_loggedin 03/09/2022 14:31:22 45900110 \course_module_viewed\resource\38297 03/09/2022 14:33:05 45900113 \course_module_viewed\resource\38298 03/09/2022 14:33:52 45900114 \course_module_viewed\forum\116492 03/09/2022 14:34:08 45900137 \post_created\forum_posts\157270 03/09/2022 14:34:50 45900141 \course_module_viewed\quiz\35804 03/09/2022 14:35:00 45900142 \attempt_started\quiz_attempts\125031 03/09/2022 14:35:03 45900152 \course_module_viewed\assign\93883 03/09/2022 14:36:20 45900199 \submission_created\assignsubmission_file\727797 03/09/2022 14:37:02 45900200 \user_loggedout 03/09/2022 14:38:10 501 45962575 \user_loggedin 03/09/2022 20:00:02 45962579 \course_module_viewed\resource\38297 03/09/2022 20:02:01 45962597 \user_loggedout 03/09/2022 20:38:10 Learner activity performance is analyzed from event logs based on frequency objects, timestamps [44], and activity [22] then five relevant activity performance variables are determined, namely 1) frequency of attendance, 2) duration of attendance, 3) frequency of access material, 4) number of posts to forum, and 5) activity conformance. Figure 3 presents the stages of activity performance analysis in each LLO: • Preparation: The preparation stage begins with preparing the event log from the LMS, followed by extraction and preprocessing. Data extraction aims to retrieve data as needed, namely the scope of LLO, timestamp range, activity, and other relevant attributes, such as username. Furthermore, the extracted data is preprocessed, in the form of data structuring by sorting based on username and timestamp, activity labeling, and data formatting for further processing needs. • Exploration and measurement: The extracted and formatted event log data is then explored and measured to obtain activity performance values. Figure 3 shows two techniques to obtain performance values from the five activity performance variables. First, to get performance values from frequency of attendance, duration of attendance, frequency of access material, and number of posts to the forum, the timestamps and activity attributes are calculated using a statistical approach, such as aggregating, summing, calculating differences, etc. Second, to obtain performance values from the activity conformance variable, activity attributes are explored and measured using process mining techniques with an alignment-based conformance checking algorithm [27]. Exploration in the form of making process models comes from activity designs and process models comes from actual learner activities with Petri Net. From the two process models, measurement is then carried out by calculating the fitness value which indicates the activities’ conformance, using an alignment-based approach. • Categorization of performance levels: After the performance value is obtained, the performance level is categorized into High, Medium, and Low [52, 53]. Before categorizing, a value is assigned to each variable as a performance standard [47, 53], as follows: HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 579 o Frequency of attendance: The frequency of attendance collection method calculates the frequency of login-logout sessions in e-learning by learners in an LLO. The default performance value of this variable is the number of days an LLO is executed. For example, LLO 1 with a duration of 2 weeks or 14 days, as presented in Table 3, has the highest attendance frequency value of 14 and the lowest is 0. The highest and lowest attendance ranges form the basis for categorizing performance levels as High, Medium, or Low using Equations 1 and 2 [53]. Figure 3. Flowchart depicting the activity performance analysis 𝑀𝑅 = 𝑝𝑚𝑎𝑥 − 𝑝𝑚𝑖𝑛 2 (1) 𝑆𝐷 = 𝑝𝑚𝑎𝑥 − 𝑝𝑚𝑖𝑛 6 (2) where 𝑝 is activity performance, including frequency of attendance, duration of attendance, frequency of access material, number of posts to the forum, and activity conformance; 𝑝𝑚𝑎𝑥 is the highest value of a performance, 𝑝𝑚𝑖𝑛 is the lowest value of a performance; 𝑀𝑅 is the middle value of 𝑝𝑚𝑎𝑥 and 𝑝𝑚𝑖𝑛, and 𝑆𝐷 is the standard deviation with the number 6 indicating the number of areas in standard deviation. Based on the 𝑀𝑅 and 𝑆𝐷 values, then the level categorization is carried out based on Table 7 [53]. Table 8 presents each performance level’s five activity performance variables’ standard values. Table 7. Categorization activity performance level. Performance level Criteria High 𝑝(𝑖) ≥ (𝑀𝑅 + 𝑆𝐷) Medium (𝑀𝑅 − 𝑆𝐷) < 𝑝(𝑖) < (𝑀𝑅 + 𝑆𝐷) Low 𝑝(𝑖) ≤ (𝑀𝑅 − 𝑆𝐷) HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 580 o Duration of attendance: The attendance duration collection method calculates the total time of all learner login- logout sessions during an LLO. The standard performance values of this variable are calculated based on the course credit time value, where the course credit is 170 minutes/week [2]. If this study course has a credit score of 2, then the highest duration of attendance is two credits × 170 minutes × 2 weeks or 680 minutes, and the lowest duration is 0. The duration score is then categorized into High, Medium, or Low performance levels using Equations 1 and 2. o Frequency of access material: The material access frequency collection method counts the number of accesses to video material and PDF documents during the implementation of an LLO. The standard performance value of this variable is calculated based on the number of days of LLO duration. For example, for an LLO with a period of 2 weeks or 14 days, the highest frequency of access to material is 14, and the lowest is 0. The value of frequency of access to material is further categorized into High, Medium, or Low performance levels using Equations 1 and 2. Table 8. Performance standard value of variable activity performance Performance variables Level Standard value Frequency of attendance High 𝑝(𝑖) > 9.3 Medium 4.7 ≤ 𝑝(𝑖) ≤ 9.3 Low 𝑝(𝑖) < 4.7 Duration of attendance High 𝑝(𝑖) > 453.3 Medium 226.7 ≤ 𝑝(𝑖) ≤ 453.3 Low 𝑝(𝑖) < 226.7 Frequency of access material High 𝑝(𝑖) > 9.3 Medium 4.7 ≤ 𝑝(𝑖) ≤ 9.3 Low 𝑝(𝑖) < 4.7 Number of posting to forum High 𝑝(𝑖) ≥ 3 Medium 1 − 2 Low 0 Activity conformance High 𝑝(𝑖) > 0.7 Medium 0.3 ≤ 𝑝(𝑖) ≤ 0.7 Low 𝑝(𝑖) < 0.3 o Number of posts to the forum: The collection method of the posts to the forum is calculated from the number of opinions learners posted in the forums during the duration of an LLO. The teacher sets this performance standard, where posting performance is assessed as High if the number of posts is ≥ 3, Medium if the number of posts is 1 − 2, and Low if never posted. The value of this performance standard is presented in Table 8. o Activity conformance: The conformance of the learner’s actual activities with the design activities of Table 3 is detected using alignment-based conformance checking [27, 46]. Detection begins with creating an event logs process model and a design process model using Petri Net. Furthermore, activity conformance calculation is carried out from two process models to obtain fitness values in the range [0,1]. Fitness values are categorized into High, Medium, or Low levels using Equations 1 and 2. The calculation of conformance of activities with the alignment-based conformance checking technique in this study is as follows [46]: Definition 1: Event Log (𝑳) 𝑇 is a set of learning activities, 𝜎𝜖𝑇∗ is an event trace, i.e., series of learning activity identifiers, 𝐿𝑠 ⊆ 𝑇 ∗ is an event log, i.e., multi set of event traces. 𝐿𝑠 are the event logs of the 𝑠 learner. Learning activities are identified by a single character, i.e., a = login, b = view video material, c = view PDF material, d = view forum, e = post forum, f = view quiz, g = submitted quiz, h = view assignment, i = submitted assignment, and j = logout. 𝑇 = {𝑎, 𝑏, 𝑐, 𝑑, 𝑒, 𝑓, 𝑔, ℎ, 𝑖, 𝑗} (3) 𝜎 = {𝑎, 𝑏, 𝑐, 𝑗} (4) 𝐿1 = [〈𝑎, 𝑏, 𝑑, 𝑖, 𝑗〉, 〈𝑎, 𝑏, 𝑑, 𝑖, 𝑗〉, 〈𝑎, 𝑔, ℎ, 𝑗〉] (5) HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 581 Definition 2: Petri Net A Petri Net is a triplet 𝑁 = (𝑃, 𝑇, 𝐹), where 𝑃 is a finite set of places, 𝑇 is a finite set of transitions such that 𝑃 ∩ 𝑇 = ∅⋀𝐹 ⊆ (𝑃 × 𝑇)⋃(𝑇 × 𝑃) is a set of directed arcs (flow relation). A marked Petri Net is a pair (𝑁,𝑀), where 𝑁 = (𝑃, 𝑇, 𝐹) is a Petri Net and where 𝑀 ∈ 𝛽(𝑃) is a multi-set over 𝑃 denoting the marking of the net. The denotation of the set of all marked Petri nets is 𝑁. If 𝑇 is a set of learning activities, 𝜎𝜖𝑇∗ is an event trace with length 𝑛 at 𝑇, then the event net of 𝜎 is a Petri Net 𝑁 = (𝑃, 𝑇, 𝐹) where: 𝑃 = {𝑝𝑗|1 ≤ 𝑗 ≤ 𝑛 + 1} 𝑇 = {𝑡𝑗|1 ≤ 𝑗 ≤ 𝑛} (6) 𝐹: (𝑃 × 𝑇) ∪ (𝑇 ∪ 𝑃) → 𝑁 (7) With; 𝐹(𝑝𝑗 , 𝑡𝑗) = 1 , ∀ 1 ≤ 𝑗 ≤ 𝑛, 𝑝𝑗 ∈ 𝑃, 𝑡𝑗 ∈ 𝑇 (8) 𝐹(𝑡𝑗, 𝑝𝑗+1) = 1 , ∀ 1 ≤ 𝑗 ≤ 𝑛, 𝑝𝑗 ∈ 𝑃, 𝑡𝑗 ∈ 𝑇 (9) 𝐹(𝑥, 𝑦) = 0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒 (10) Figure 4 presents the process model of design activities for this study in Petri Net according to learning activities Table 4. Figure 4. Process models of learning activities in Petri Nets Definition 3: Alignment-based conformance checking Let 𝑇 be a set of learning activities, 𝜎 ∈ 𝑇∗ be a trace of length n over T. Let 𝐿 be transitions of an event net, 𝑀 be transitions of a Petri Net, (𝑙, 𝑚) be a movement sequence of alignment 𝛾 where (𝑙, 𝑚) is one of the following movements: • 𝑚𝑜𝑣𝑒 𝑜𝑛 log 𝑖𝑓 𝑙 ∈ 𝐿 and 𝑚 = ≫, • 𝑚𝑜𝑣𝑒 𝑜𝑛model 𝑖𝑓 𝑙 =≫ and 𝑚 ∈ 𝑀, • 𝑠𝑦𝑛𝑐ℎ𝑟𝑜𝑛𝑜𝑢𝑠 𝑚𝑜𝑣𝑒 𝑖𝑓 𝑙 ∈ 𝐿 and 𝑚 ∈ 𝑀, • illegal move 𝑖𝑓 𝑙 =≫ and 𝑚 = ≫. The distance function 𝛿 is a quality value of an alignment that associates costs to moves in the alignment: • 𝑖𝑓 𝑙 ∈ 𝐿 and 𝑚 =≫, then 𝛿(𝑙,𝑚) is the cost of move 𝑙 in the log, • 𝑖𝑓 𝑙 =≫ and 𝑚 ∈ 𝑀, then 𝛿(𝑙,𝑚) is the cost of move 𝑚 in the model, • 𝑖𝑓 𝑙 ∈ 𝐿 and 𝑚 ∈ 𝑀, then 𝛿(𝑙,𝑚) is the cost of move 𝑙 in the log and move 𝑚 in the model. Moves in just the log or model have cost 1. There is a particular case for the move-on model when the transition is invisible, in that case, we use cost 0. The distance of the whole alignment is calculated as the sum of the costs that appear in the alignment. The distance function 𝛿 associates high costs to move where both log and model make a move but disagree on the learning activities. Definition 4: Fitness of alignment Let 𝐿 be an event log, 𝜎 be a trace over 𝐿, 𝑁 be a Petri Net, 𝛿 ((𝛾𝑜𝑝𝑡 𝑁 (𝜎))) is an optimal alignment of trace 𝜎 on N, 𝛿 ((𝛾𝑤𝑜𝑟𝑠𝑡 𝑁 (𝜎))) is the worst alignment of trace 𝜎 on N. The fitness is defined as follows: 𝑓𝑖𝑡𝑛𝑒𝑠𝑠 (𝜎, 𝑁) = 1 − 𝛿 (𝛾𝑜𝑝𝑡 𝑁 (𝜎)) 𝛿(𝛾𝑤𝑜𝑟𝑠𝑡 𝑁 (𝜎)) (11) 𝛿(𝛾𝑤𝑜𝑟𝑠𝑡 𝑁 (𝜎)) is a sum of the event activities and the shortest path through the Petri Net. According to code Table 9, the fitness for a trace is calculated as: 𝑓𝑖𝑡𝑛𝑒𝑠𝑠 (𝜎, 𝑁) = 1 − 𝑐𝑜𝑠𝑡 𝑏𝑤𝑐 (12) HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 582 with 𝑏𝑤𝑐 is the sum of the length of the trace and the length of the shortest path in the model taking from the initial marking to the final marking. Based on Figure 3, activity performance processing for variables related to frequency and duration uses code Table 9 line 1 to 38. The event log data is extracted, preprocessed, and formatted to obtain a dataset with several attributes suitable for frequency and duration calculations: Case ID, activity, timestamps, and username. The processing of the activity conformance variables uses code Table 9 line 39 to 72. From the event log data that has been extracted, preprocessed, and formatted are then made to create a process model for the design and actual activities of the learner. Furthermore, alignment-based fitness calculations are carried out in both models. Table 9. Code for analysis of object frequency, duration, and activity conformance Line Code Line Code 1 # Analysis frequency and duration 38 duration_access_ST001 = minutes 2 # Convert event log to data frame 39 # Analysis activity conformance. Calculate fitness 3 import io 40 # Convert event log to data frame 4 import pandas as pd 41 import io 5 dt_learner = pd.read_csv(r'performa.csv', sep = ',', low_memory = 'False') 42 import pandas as pd 6 dt_learner 43 dt_student = pd.read_csv(r'Calc_align.csv', sep = ',', low_memory = 'False') 7 # Filter each learner 44 dt_student 8 dt_ST001 = dt_learner[dt_learner["firstname"] == "ST001"] 45 # Filter each learner 9 # Selection of relevant attributes 46 dt_ST001 = dt_student[dt_student["firstname"] == "ST001"] 10 selected_dt_ST001 = dt_ST001[[‘session’,’labelling’,’date’]] 47 # Selection of relevant attributes 11 # Convert data frame to log 48 selected_dt_ST001 = dt_ST001[[‘session’, ‘labelling’, ‘date’]] 12 import pm4py as pm4 49 # Convert data frame to log 13 from pm4py.objects.conversion.log import converter as log_converter 50 data_ideal = data_mahasiswa[data_mahasiswa["firstname"] == "STIdeal"] 14 renamed_dt_ST001 = selected_dt_ST001.rename(columns= {'session': 'case:concept:name', 'labelling':'concept:name', 'date': 'time:timestamp'}) 51 renamed_dt_ST001 = selected_dt_ST001.rename(columns= {'session': 'case:concept:name', 'labelling':'concept:name', 'date': 'time:timestamp'}) 15 start = pm4.get_start_activities(renamed_dt_ST001) 52 start = pm4.get_start_activities(renamed_dt_ST001) 16 df_start_activities = pm4.filter_start_activities (renamed_dt_ST001, ['A']) 53 df_start_activities = pm4.filter_start_activities(renamed_dt_ST001, ['A']) 17 end_activities = pm4.get_end_activities(df_start_activities) 54 end_activities = pm4.get_end_activities(df_start_activities) 18 df_filtered_end_dt_ST001 = pm4.filter_end_activities (df_start_activities, end_activities) 55 df_filtered_end_dt_ST001 = pm4.filter_end_activities (df_start_activities, end_activities) 19 log_ST001 = log_converter.apply(df_filtered_end_dt_ST001) 56 log_ST001 = log_converter.apply(df_filtered_end_dt_ST001) 20 # Filter on trace 57 from pm4py.algo.filtering.log.variants import variants_filter 21 from pm4py.algo.filtering.log.variants import variants_filter 58 from pm4py.statistics.traces.generic.log import case_statistics 22 from pm4py.statistics.traces.generic.log import case_statistics 59 variants_count = case_statistics.get_variant_statistics(log_ST001) 23 variants_count = case_statistics.get_variant_statistics (log_ST001) 60 variants_count = sorted(variants_count, key=lambda x: x['count'], reverse=True) 24 variants_count = sorted (variants_count, key=lambda x: x['count'], reverse=True) 61 variants_count = case_statistics.get_variant_statistics(log_ST001) 25 variants_count 62 variants_count 26 # Calculate the frequency of activity 63 from pm4py.algo.filtering.log.variants import variants_filter 27 frequency_label_ST001 = dt_ST001['labelling'].value_counts() 64 from pm4py.statistics.traces.generic.log import case_statistics 28 print(frequency_label_ST001) 65 # Discovery log design 29 # Calculate the duration of learner activity 66 net, initial_marking, final_marking = pm4.discover_petri_net_inductive (log_design) 30 from datetime import datetime 67 # Build simulated log design 31 start = datetime.strptime(“hh:mm:ss”, “%H:%M:%S”) 68 Simulated_log = pm4.play_out(net, initial_marking, final_marking) 32 end = datetime.strptime(“hh:mm:ss”, “%H:%M:%S”) 69 from pm4py.algo.conformance.alignments.edit_distance import algorithm as logs_alignments 33 difference = end – start 70 # Find alignment 34 seconds = difference.total_seconds( ) 71 alignments = logs_alignments.apply(log_ST001, simulated_log) 35 minutes = seconds / 60 72 alignments 37 hours = second / (60 * 60) HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 583 3.3. Unification of Assessment Data with 2-tuple Fuzzy Linguistics Data related to ability level is processed as follows: • Log records provide disciplinary performance assessment data. This performance is calculated from the delays in submitting assignments in each LLO. The teacher determines this performance value, as presented in Table 10. The quiz assessment data represents elements of knowledge, while the e-rubric assessment data represents skills elements. • Numerical assessment data from recorded logs and quizzes, and e-rubric in linguistics are then unified using 2-tuple fuzzy linguistic to obtain ability levels, as shown in Figure 5. Table 10. Discipline performance standard Delays in submitting quizzes and assignments (hours) Standard value 0 100 1-6 75 7-12 70 13-18 65 19-24 60 > 24 55 Figure 5. Flowchart depicting 2-tuple fuzzy linguistic. HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 584 The proposed assessment using linguistic data required a linguistic data representation approach. The actual grading system in university of this study [54] become the basis for determining the linguistic data representation in this study, as presented Equation 13. 𝑆 = {𝑠0, 𝑠1, 𝑠2, 𝑠3, 𝑠4, 𝑠5, 𝑠6, 𝑠7, 𝑠8} (13) where, 𝑆 is a linguistic term set that spread across 9 cardinals. The semantics are defined using a fuzzy membership function and described by a triangular fuzzy number (TFN). The linguistic and semantic set is illustrated in Table 11. Table 11. The linguistics sets and their semantics. Symbol Abbreviation Linguistic Term Triangular Fuzzy Number VI 𝑠0 Very Insufficient (0, 0, 0.5) I 𝑠1 Insufficient (0, 0.5, 0.55) PA 𝑠2 Partially Acceptable (0.5, 0.55, 0.6) A 𝑠3 Acceptable (0.55, 0.6, 0.65) S 𝑠4 Satisfactory (0.6, 0.65, 0.7) G 𝑠5 Good (0.65, 0.7, 0.75) VG 𝑠6 Very Good (0.7, 0.75, 0.8) E 𝑠7 Excellent (0.75, 0.8, 1) OS 𝑠8 Out Standing (0.8, 1, 1) According to Figure 5, the unification of numerical and linguistic data using 2-tuple fuzzy linguistics is carried out through preprocessing, transformation, and aggregation [51], as follows: • Preprocessing: In the preprocessing stage, the numerical assessment data [0,10] or [0,100] is converted into [0,1]. • Transformation: Data in linguistic 𝑥 ∈ 𝑆 = {𝑠0, ⋯ , 𝑠𝑔} where 𝑔 is the number of linguistic terms, according to Table 11, and data in numeric 𝑥 ∈ [0,1] transformed to 2-tuple linguistic values with the following steps: o Numerical data: Transform 𝑥 ∈ [0,1] to a 2-tuple linguistic value as follows: ➢ Converting 𝑥 into a fuzzy set in 𝑆 according to Table 11 with the function 𝜏 and function 𝜃. 𝜏: [0,1] → 𝐹(𝑆) 𝜏(𝑥) = {(𝑠0, 𝜃0),⋯ , (𝑠𝑔 , 𝜃𝑔)}, 𝑠𝑖𝜖𝑆 𝑎𝑛𝑑 𝜃𝑖 ∈ [0,1] (14) With; 𝜃𝑖 = 𝜇𝑠𝑖(𝑥) = { 0 𝑖𝑓 𝑥 ∉ 𝑠𝑢𝑝𝑝𝑜𝑟𝑡 (𝜇𝑠𝑖(𝑥)) 𝑥 − 𝑎𝑖 𝑏𝑖 − 𝑎𝑖 𝑖𝑓 𝑎𝑖 ≤ 𝑥 < 𝑏𝑖 1 𝑖𝑓 𝑏𝑖 ≤ 𝑥 < 𝑑𝑖 𝑐𝑖 − 𝑥 𝑐𝑖 − 𝑑𝑖 𝑖𝑓 𝑑𝑖 ≤ 𝑥 ≤ 𝑐𝑖 (15) The value of (𝑎𝑖 , 𝑏𝑖 , 𝑐𝑖 , 𝑑𝑖) are the lower and upper limits of 𝑥 according to the TFN value in Table 11. ➢ Furthermore, a search for numerical values that represent information from the fuzzy set [0, 𝑔] is carried out through the function 𝜒 (16) based on the results of the representation of the numerical value 𝑥 ∈ [0,1] in the linguistic set 𝑆 = {𝑠0, ⋯ , 𝑠𝑔} using Equations 14 and 15. 𝜒: 𝐹(𝑆) → [0, 𝑔] 𝜒({(𝑠𝑗 , 𝜃𝑗)|𝑗 = 0,⋯ , 𝑔}) = ∑ 𝑗𝜃𝑗 𝑔 𝑗=0 ∑ 𝜃𝑗 𝑔 𝑗=0 = 𝛽 (16) ➢ Transforming the value of 𝛽 into a linguistic 2-tuple with Equation 17. ∆: [0, 𝑔] → 𝑆 × [−0.5,0.5] ∆(𝛽) = (𝑠𝑖 , 𝛼) 𝑤𝑖𝑡ℎ { 𝑠𝑖 , 𝑖 = 𝑟𝑜𝑢𝑛𝑑(𝛽) 𝛼 = 𝛽 − 𝑖, 𝛼 ∈ [−0.5,0.5) (17) HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 585 with 𝑠𝑖 ∈ 𝑆, 𝑖 = 𝑟𝑜𝑢𝑛𝑑(𝛽), 𝛼 = 𝛽 − 𝑖, and 𝛼 ∈ [−0.5,0.5), 𝑟𝑜𝑢𝑛𝑑 is the rounding operation, 𝑠𝑖 is the index label closest to 𝛽, and 𝛼 is the symbolic translation value. o Linguistic data: The linguistic term transformation 𝑠𝑖 ∈ 𝑆 = {𝑠0, ⋯ , 𝑠𝑔} in 2-tuple linguistic equivalence is obtained by the function Φ Equation 18. Φ: 𝑆 → (𝑆 × [−0.5,0.5)) Φ(𝑠𝑖) = (𝑠𝑖 , 0), 𝑠𝑖 ∈ 𝑆 (18) • Aggregation: The aggregation stage of the 2-tuple linguistic values includes two ways using Equations 19 and 21. o Aggregating 2-tuple linguistic values in each assessment group, i.e., e-rubric, with arithmetic mean begins with defining the numerical equivalent 𝛽 ∈ [0, 𝑔] of 2-tuple linguistic values with Equation 19. ∆−1: 𝑆 × [−0.5,0.5] → [0, 𝑔] ∆−1(𝑠𝑖 , 𝛼) = 𝑖 + 𝛼 = 𝛽 (19) The function ∆−1 is the inverse function of the function ∆ of the equation 𝑆. For example, 𝑥 = {(𝑠1, 𝛼1), (𝑠2, 𝛼2),⋯ , (𝑠𝑛 , 𝛼𝑛)} is a 2-tuple linguistic set, then arithmetic mean is obtained by Equation 20. (�̅�, �̅�) = ∆( 1 𝑛 ∑∆−1(𝑠𝑗 , 𝛼𝑗) 𝑛 𝑗=1 ) , �̅� ∈ 𝑆, �̅� ∈ [− 0.5,0.5) (20) with �̅� is the mean value of 𝑠. o Aggregating 2-tuple linguistic values with weight values 𝑊 = {𝑤1, 𝑤2, ⋯ , 𝑤𝑖}, where 𝑖 is the number of indicators in all LLOs. If 𝑊 is the associated weight, then the average weight of the 2-tuple �̅�𝑤 is as Equation 21. �̅�𝑊 = ∆( ∑ ∆−1𝑛 𝑖=1 (𝑠𝑖 , 𝛼𝑖) ∙ 𝑤𝑖 ∑ 𝑤𝑖 𝑛 𝑖=1 ) (21) The weight 𝑊is determined by the teacher’s preferences [51]. 3.4. Rule-Based Learning Recommendation The value of 𝑠 in the ability level (𝑠, 𝛼), and the level of the five activity performance variables is a fact that becomes recommendation input. The input facts are processed into recommendations using the rule base Equation 22. 𝐼𝐹 𝐹𝑎𝑐𝑡 1 𝐴𝑁𝐷 𝐹𝑎𝑐𝑡 2 𝐴𝑁𝐷…𝐴𝑁𝐷 𝐹𝑎𝑐𝑡𝑛 𝑇𝐻𝐸𝑁 𝐶𝑜𝑛𝑐𝑙𝑢𝑠𝑖𝑜𝑛 (22) Based on Equation 22, there are six facts as input, including Fact 1 is the value of 𝑠 level of ability, while Fact 2, Fact 3, Fact 4, Fact 5, and Fact 6 are the performance level of the variable frequency of attendance, duration of attendance, frequency of access material, number of posts to forum, and activity conformance, as presented in Table 12. The conclusion provides two recommendations. First, recommendations regarding implementing LLOs, whether it was Succeed or Failed. This recommendation is determined based on linguistic values 𝑠. In this study, university policy stipulates Succeed if 𝑠 ∈ {𝑠3, 𝑠4 , 𝑠5, 𝑠6, 𝑠7, 𝑠8}, and stipulates Failed if 𝑠 ∈ {𝑠0, 𝑠1, 𝑠2}. If the result is Failed, the learner must repair or repeat the LLO. Second, recommendations in the form of directions on the achieved activity performance, are presented in Table 12. 4. Result and Discussion This section presents the results of implementing the system in the Basic Programming course with two credits. As a data sample, 20 learners were taken during the implementation of LLO 2 within two weeks. 4.1. Data Introduction This assessment system produces two datasets related to ability level and activity performance in each LLO implementation. Table 13 presents datasets related to ability levels, and Table 14 is related to activity performance. The assessment data to determine the ability level consisted of a knowledge score from the quiz, four skill dimension scores from the e-rubric, and a discipline attitude score from the logs recording. The four skills dimensions from e-rubric include a) the ability to construct correct program flow, b) the ability to use and write program elements, c) the ability to apply composition techniques in program structure and syntax, and d) the program works and meets all specifications, as presented in Figure 2. HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 586 Table 12. Activity performance direction Performance variables Level Direction Frequency of attendance High High attendance frequency. Keep it up! Medium Medium attendance. Increase attendance so that learning outcomes are better. Low Low frequency of attendance, must access e-learning more often! Duration of attendance High High duration of attendance. Keep it up! Medium Medium duration of attendance. Increase the time to access e-learning for better learning outcomes! Low Low access duration! Allocate more time to study in e-learning! Frequency of access material High High material access. Keep it up! Medium Medium material access. Increase the frequency of access to video and PDF materials for even better learning outcomes! Low Low material access. Open and read the material provided so you can understand the material presented! Number of posts to forum High High forum participation. Keep it up! Medium Participation in forums is Medium. Increase involvement in discussions! Low Forum participation is Low. Join the forum and share your opinion! Activity Conformance High Conformance of the activities against the teacher’s directions is High. Keep it up! Medium Conformance of activities with the direction of the teacher is Medium. Improve obedience to the teacher’s guides for even better learning outcomes! Low Conformance of the activities with the teacher’s direction is Low. Pay attention to the teacher’s guides and do the activities in the order specified! Table 13. Level ability data Learner Quiz 1st dimension of e-rubric 2nd dimension of e-rubric 3rd dimension of e-rubric 4th dimension of e-rubric Value of discipline attitude 𝐿1 100 Very Good Excellent Good Good 100 𝐿2 95 Excellent Very Good Good Good 100 𝐿3 100 Acceptable Very Good Satisfactory Very Good 100 𝐿4 100 Very Good Good Good Acceptable 100 𝐿5 100 Very Good Excellent Very Good Excellent 100 𝐿6 100 Very Good Excellent Very Good Good 100 𝐿7 75 Out Standing Good Good Good 100 𝐿8 60 Good Good Good Satisfactory 100 𝐿9 100 Good Good Acceptable Acceptable 75 𝐿10 35 Very Insufficient Very Insufficient Very Insufficient Very Insufficient 100 𝐿11 100 Satisfactory Good Satisfactory Good 60 𝐿12 0 Very Insufficient Very Insufficient Very Insufficient Very Insufficient 0 𝐿13 0 Very Good Good Good Very Good 0 𝐿14 100 Very Good Very Good Very Good Excellent 100 𝐿15 100 Very Good Good Satisfactory Good 100 𝐿16 100 Very Good Very Good Very Good Very Good 100 𝐿17 100 Very Good Good Acceptable Good 100 𝐿18 100 Very Good Good Acceptable Good 100 𝐿19 100 Satisfactory Good Good Good 100 𝐿20 73 Satisfactory Good Very Good Good 100 In Table 13, for example, Learner 9 (𝐿9) scored 100 for assessing the elements of knowledge by quiz. For the skills assessment by e-rubric, 𝐿9 scores Good on dimension 1, Good on dimension 2, Acceptable on dimension 3, and Acceptable on dimension 4. For assessing the aspect of disciplinary attitude, 𝐿9 scores of 75 mean there was a delay in submitting tasks in 1-6 hours, as presented in Table 10. Table 14 presents activity performance data, including frequency of attendance (𝑝1), duration of attendance (𝑝2), frequency of access material (𝑝3), number of posts to forum (𝑝4), activity conformance (𝑝5). In Table 14, for example, HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 587 during the implementation of LLO 2, 𝐿1 accessed the course 18 times, the duration of attendance was 869 minutes, the frequency of accessing video materials and PDF documents was one time, posted an opinion in the forum one time, and the fitness value of activity conformance was 0.667. It differs from 𝐿12 where all performance data, including 𝑝1, 𝑝2, 𝑝3, 𝑝4, and 𝑝5, have a value of 0. It means that during the implementation of LLO 2, 𝐿12 did not carry out learning activities in e-learning. Table 14. Data related to activity performance variables Learner 𝒑𝟏 𝒑𝟐 𝒑𝟑 𝒑𝟒 𝒑𝟓 𝐿1 18 869 1 1 0.667 𝐿2 11 1011 6 0 0.571 𝐿3 4 286 2 0 0.824 𝐿4 4 374 1 0 0.778 𝐿5 16 621 12 1 0.824 𝐿6 10 198 6 0 0.824 𝐿7 7 95 2 0 0.462 𝐿8 11 712 3 0 0.632 𝐿9 12 643 5 0 0.571 𝐿10 6 516 3 0 0.462 𝐿11 6 922 1 0 0.625 𝐿12 0 0 0 0 0.000 𝐿13 5 513 4 0 0.571 𝐿14 4 159 1 0 0.667 𝐿15 9 1883 7 0 0.625 𝐿16 5 408 2 0 0.571 𝐿17 4 168 1 0 0.571 𝐿18 7 403 0 0 0.750 𝐿19 4 641 2 0 0.667 𝐿20 5 1544 0 0 0.625 4.2. Result 4.2.1. Level ability • Unification on e-rubric [54]: Based on Table 3, in LLO-2 there is one e-rubric for the assessment instrument indicator 2.2., which has 4 dimensions of assessment. If 𝐿 = {𝐿1, ⋯ , 𝐿20} is the number of learners, and 𝐷 = {𝐷1, ⋯ , 𝐷4} is the number of assessment dimensions in the e-rubric. o Preferences by teachers in e-rubrics 2.2 are presented in the decision matrix 𝑅𝐵 = (𝑟𝑖𝑗)𝑚ℎ where 𝑟𝑖𝑗 ∈ 𝑆𝑖 = {𝑠0, 𝑠1, ⋯ , 𝑠8} with 𝐵 = {𝐵2.2} is an indicator in LLO 2. Preferences of indicator 2.2 by teachers for 20 learners as presented Equation 23. For example, based on Table 13, the teacher’s preferences for 𝐿1 in e-rubric 2.2 are Very Good for dimension 1, Excellent for dimension 2, Good for dimension 3, and Good for dimension 4, which is represented by the symbols 𝑉𝐺, 𝐸, 𝐺 and 𝐺 as presented Equation 23. Meanwhile, for 𝐿12, the teacher’s preference for all dimensions of the e-rubric 2.2 is Very Insufficient, represented by the symbol 𝑉𝐼. o Each element in Equation 23 transformed into a 2-tuple linguistic using Equation 18 to obtain Equation 24, with 𝑅𝐿2𝑇 2.2 is linguistic 2-tuples (L2T) matrices, as presented Equation 24. For example, the teacher’s preferences for 𝐿1 in Equation 23 are 𝑉𝐺 for 𝐷1, 𝐸 for 𝐷2, 𝐺 for 𝐷3, and 𝐺 for 𝐷4, the symbol of these preferences are transformed in 2-tuple linguistic form into (𝑉𝐺, 0), (𝐸, 0), (𝐺, 0), and (𝐺, 0), as presented in Equation 24. o The 2-tuple linguistic of each matrix in are aggregated using Equations 19 and 20. The aggregation of 2-tuple linguistic values for e-rubric is represented by Equation 25. For example, the 2-tuple linguistic form for 𝐿1 is (𝑉𝐺, 0) for 𝐷1, (𝐸, 0) for 𝐷2, (𝐺, 0) for 𝐷3, and (𝐺, 0) for 𝐷4, each 2-tuple linguistic has a numerical representation value, namely 6 for 𝑉𝐺, 7 for 𝐸, and 5 for 𝐺, then using the inverse function and arithmetic mean Equation 20, the numeric value is 5.75 or (𝑉𝐺,−0.25) in 2-tuple linguistic, as presented in Equation 25. HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 588 𝑅2.2 = 𝐷1 𝐷2 𝐷3 𝐷4 𝐿1 𝐿2 𝐿3 𝐿4 𝐿5 𝐿6 𝐿7 𝐿8 𝐿9 𝐿10 𝐿11 𝐿12 𝐿13 𝐿14 𝐿15 𝐿16 𝐿17 𝐿18 𝐿19 𝐿20 [ 𝑉𝐺 𝐸 𝐺 𝐺 𝐸 𝑉𝐺 𝐺 𝐺 𝐴 𝑉𝐺 𝑆 𝑉𝐺 𝑉𝐺 𝐺 𝐺 𝐴 𝑉𝐺 𝐸 𝑉𝐺 𝐸 𝑉𝐺 𝐸 𝑉𝐺 𝐺 𝑂𝑆 𝐺 𝐺 𝐺 𝐺 𝐺 𝐺 𝑆 𝐺 𝐺 𝐴 𝐴 𝑉𝐼 𝑉𝐼 𝑉𝐼 𝑉𝐼 𝑆 𝐺 𝑆 𝐺 𝑉𝐼 𝑉𝐼 𝑉𝐼 𝑉𝐼 𝑉𝐺 𝐺 𝐺 𝑉𝐺 𝑉𝐺 𝑉𝐺 𝑉𝐺 𝐸 𝑉𝐺 𝐺 𝑆 𝐺 𝑉𝐺 𝑉𝐺 𝑉𝐺 𝑉𝐺 𝑉𝐺 𝐺 𝐴 𝐺 𝑉𝐺 𝐺 𝐴 𝐺 𝑆 𝐺 𝐺 𝐺 𝑆 𝐺 𝑉𝐺 𝐺 ] (23) 𝑅𝐿2𝑇 2.2 = 𝐷1 𝐷2 𝐷3 𝐷4 𝐿1 𝐿2 𝐿3 𝐿4 𝐿5 𝐿6 𝐿7 𝐿8 𝐿9 𝐿10 𝐿11 𝐿12 𝐿13 𝐿14 𝐿15 𝐿16 𝐿17 𝐿18 𝐿19 𝐿20 [ (𝑉𝐺, 0) (𝐸, 0) (𝐺, 0) (𝐺, 0) (𝐸, 0) (𝑉𝐺, 0) (𝐺, 0) (𝐺, 0) (𝐴, 0) (𝑉𝐺, 0) (𝑆, 0) (𝑉𝐺, 0) (𝑉𝐺, 0) (𝐺, 0) (𝐺, 0) (𝐴, 0) (𝑉𝐺, 0) (𝐸, 0) (𝑉𝐺, 0) (𝐸, 0) (𝑉𝐺, 0) (𝐸, 0) (𝑉𝐺, 0) (𝐺, 0) (𝑂𝑆, 0) (𝐺, 0) (𝐺, 0) (𝐺, 0) (𝐺, 0) (𝐺, 0) (𝐺, 0) (𝑆, 0) (𝐺, 0) (𝐺, 0) (𝐴, 0) (𝐴, 0) (𝑉𝐼, 0) (𝑉𝐼, 0) (𝑉𝐼, 0) (𝑉𝐼, 0) (𝑆, 0) (𝐺, 0) (𝑆, 0) (𝐺, 0) (𝑉𝐼, 0) (𝑉𝐼, 0) (𝑉𝐼, 0) (𝑉𝐼, 0) (𝑉𝐺, 0) (𝐺, 0) (𝐺, 0) (𝑉𝐺, 0) (𝑉𝐺, 0) (𝑉𝐺, 0) (𝑉𝐺, 0) (𝐸, 0) (𝑉𝐺, 0) (𝐺, 0) (𝑆, 0) (𝐺, 0) (𝑉𝐺, 0) (𝑉𝐺, 0) (𝑉𝐺, 0) (𝑉𝐺, 0) (𝑉𝐺, 0) (𝐺, 0) (𝐴, 0) (𝐺, 0) (𝑉𝐺, 0) (𝐺, 0) (𝐴, 0) (𝐺, 0) (𝑆, 0) (𝐺, 0) (𝐺, 0) (𝐺, 0) (𝑆, 0) (𝐺, 0) (𝑉𝐺, 0) (𝐺, 0) ] (24) • Unification of all assessment data: LLO-2 has two numerical assessment data from indicators 2.1, and 2.3, and one linguistic assessment data from indicator 2.2. The variety of assessment data with a combination of numerical and linguistic in this study is different from previous studies, such as Wardoyo & Yuniarti (2020) [9] and Sudaryono et al. (2020) [36], which only used numerical data, and Azimjonov (2016) [8], which only used linguistic data. Unification of assessment data in numerical and linguistic form is carried out in the following steps: o Preferences by teachers are presented in a decision matrix 𝑅𝐵 = (𝑟𝑖𝑗)𝑚ℎ where 𝑟𝑖𝑗 ∈ [0,1] for numeric data, 𝑟𝑖𝑗 ∈ 𝑆𝑖 = [𝑠0, 𝑠1, ⋯ , 𝑠𝑛] for linguistic data, and 𝐵 = {𝐵2} is the number of LLOs. Equation 26 presents learner ability level data using Table 13. For example, the ability level data for 𝐿1 is 100 for the assessment of knowledge elements obtained from quizzes (2.1), 100 for the assessment of attitude elements obtained from calculating late submission of quizzes and assignments (2.3), and (𝑉𝐺, −0.25) for the assessment of skills elements obtained from e-rubric (2.2), which processing using 2-tuple fuzzy linguistic, as presented Equations 23 to 25. Meanwhile, for 𝐿12, the ability level data for the elements of knowledge, skills, and attitude elements are 0, (𝑉𝐼, 0) and 0, respectively. HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 589 𝑅𝐿2𝑇 2.2 = 𝐿1 𝐿2 𝐿3 𝐿4 𝐿5 𝐿6 𝐿7 𝐿8 𝐿9 𝐿10 𝐿11 𝐿12 𝐿13 𝐿14 𝐿15 𝐿16 𝐿17 𝐿18 𝐿19 𝐿20 [ (𝑉𝐺,−0.25) (𝑉𝐺,−0.25) (𝐺, −0.25) (𝐺, −0.25) (𝐸, −0.5) (𝑉𝐺, 0) (𝑉𝐺,−0.25) (𝐺, −0.25) (𝑆, 0) (𝑉𝐼, 0) (𝐺, −0.5) (𝑉𝐼, 0) (𝑉𝐺,−0.5) (𝑉𝐺, 0.25) (𝐺, 0) (𝑉𝐺, 0) (𝐺, −0.25) (𝐺, −0.25) (𝐺, −0.25) (𝐺, 0) ] (25) 𝑅2 = 2.1 2.2 2.3 𝐿1 𝐿2 𝐿3 𝐿4 𝐿5 𝐿6 𝐿7 𝐿8 𝐿9 𝐿10 𝐿11 𝐿12 𝐿13 𝐿14 𝐿15 𝐿16 𝐿17 𝐿18 𝐿19 𝐿20 [ 100 (𝑉𝐺,−0.25) 100 95 (𝑉𝐺,−0.25) 100 100 (𝐺,−0.25) 100 100 (𝐺,−0.25) 100 100 (𝐸,−0.5) 100 100 (𝑉𝐺, 0) 100 75 (𝑉𝐺,−0.25) 100 60 (𝐺,−0.25) 100 100 (𝑆, 0) 75 35 (𝑉𝐼, 0) 100 100 (𝐺,−0.5) 60 0 (𝑉𝐼, 0) 0 0 (𝑉𝐺,−0.5) 0 100 (𝑉𝐺, 0.25) 100 100 (𝐺, 0) 100 100 (𝑉𝐺, 0) 100 100 (𝐺,−0.25) 100 100 (𝐺,−0.25) 100 100 (𝐺,−0.25) 100 73 (𝐺, 0) 100] (26) o Preprocessing is done by changing the score [0,100] to. [0,1]. For example, for 𝐿1, the numerical value 100 in elements (2.1) and (2.3) is converted to 1.0. Meanwhile, the numerical value 0 in elements (2.1) and (2.3) is converted to 0 for 𝐿12. o Numerical data is transformed to 2-tuple linguistic using Equations 14 to 17) with the result Equation 28. For example, the numerical value 1.0 in elements (2.1) for 𝐿1 is equal to (𝑂𝑆, 0), the numerical value 0.75 in elements (2.1) for 𝐿7 is equal to (𝑉𝐺, 0), and the numerical value 0.0 in elements (2.1) for 𝐿12 is equal to (𝑉𝐼, 0). o From (28), aggregation is implemented to determine the level of the learner’s ability. This aggregation considers the weight of each assessment technique based on teacher preferences 𝑊 = {0.35, 0.5, 0.15} where 0.35 is the weight for the knowledge assessment with a quiz, 0.5 for the skills assessment with e-rubric, and 0.15 for the attitude assessment. Then, the 2-tuple linguistic value of all indicators is aggregated using Equation 21. Equation (29) is an example of calculating ability level, which is accompanied by a 𝑊 weight in a 2-tuple linguistic value for 𝐿1. HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 590 𝑅2 = 2.1 2.2 2.3 𝐿1 𝐿2 𝐿3 𝐿4 𝐿5 𝐿6 𝐿7 𝐿8 𝐿9 𝐿10 𝐿11 𝐿12 𝐿13 𝐿14 𝐿15 𝐿16 𝐿17 𝐿18 𝐿19 𝐿20 [ 1.0 (𝑉𝐺,−0.25) 1.0 0.95 (𝑉𝐺,−0.25) 1.0 1.0 (𝐺, −0.25) 1.0 1.0 (𝐺, −0.25) 1.0 1.0 (𝐸, −0.5) 1.0 1.0 (𝑉𝐺, 0) 1.0 0.75 (𝑉𝐺,−0.25) 1.0 0.6 (𝐺, −0.25) 1.0 1.0 (𝑆, 0) 0.75 0.35 (𝑉𝐼, 0) 1.0 1.0 (𝐺, −0.5) 0.6 0 (𝑉𝐼, 0) 0 0 (𝑉𝐺,−0.5) 0 1.0 (𝑉𝐺, 0.25) 1.0 1.0 (𝐺, 0) 1.0 1.0 (𝑉𝐺, 0) 1.0 1.0 (𝐺, −0.25) 1.0 1.0 (𝐺, −0.25) 1.0 1.0 (𝐺, −0.25) 1.0 0.73 (𝐺, 0) 1.0 ] (27) R𝐿2𝑇 2 = 2.1 2.2 2.3 𝐿1 𝐿2 𝐿3 𝐿4 𝐿5 𝐿6 𝐿7 𝐿8 𝐿9 𝐿10 𝐿11 𝐿12 𝐿13 𝐿14 𝐿15 𝐿16 𝐿17 𝐿18 𝐿19 𝐿20 [ (𝑂𝑆, 0) (𝑉𝐺,−0.25) (𝑂𝑆, 0) (𝑂𝑆, −0.25) (𝑉𝐺,−0.25) (𝑂𝑆, 0) (𝑂𝑆, 0) (𝐺, −0.25) (𝑂𝑆, 0) (𝑂𝑆, 0) (𝐺, −0.25) (𝑂𝑆, 0) (𝑂𝑆, 0) (𝐸, −0.5) (𝑂𝑆, 0) (𝑂𝑆, 0) (𝑉𝐺, 0) (𝑂𝑆, 0) (𝑉𝐺, 0) (𝑉𝐺,−0.25) (𝑂𝑆, 0) (𝐴, 0) (𝐺, −0.25) (𝑂𝑆, 0) (𝑂𝑆, 0) (𝑆, 0) (𝑉𝐺, 0) (𝐼, −0.3) (𝑉𝐼, 0) (𝑂𝑆, 0) (𝑂𝑆, 0) (𝐺, −0.5) (𝐴, 0) (𝑉𝐼, 0) (𝑉𝐼, 0) (𝑉𝐼, 0) (𝑉𝐼, 0) (𝑉𝐺,−0.5) (𝑉𝐼, 0) (𝑂𝑆, 0) (𝑉𝐺, 0.25) (𝑂𝑆, 0) (𝑂𝑆, 0) (𝐺, 0) (𝑂𝑆, 0) (𝑂𝑆, 0) (𝑉𝐺, 0) (𝑂𝑆, 0) (𝑂𝑆, 0) (𝐺, −0.25) (𝑂𝑆, 0) (𝑂𝑆, 0) (𝐺, −0.25) (𝑂𝑆, 0) (𝑂𝑆, 0) (𝐺, −0.25) (𝑂𝑆, 0) (𝑉𝐺,− 0.4) (𝐺, 0) (𝑂𝑆, 0)] (28) �̅�𝑤 = ( 0.35 × 8.0 + 0.5 × 5.75 + 0.15 × 8.0 0.35 + 0.5 + 0.15 ) = 6.9 = (𝐸,−0.1) (29) Finally, Table 15 shows each learner’s ability more specifically in the 2-tuple linguistic. For example, 𝐿1 and 𝐿5 have Excellent abilities with different 𝛼, where 𝐿1 with (𝐸, −0.1) while 𝐿5 with (𝐸, 0.3). It means that for 𝐿1, 10% of the ability is still needed to achieve Excellent, with 𝐿5 has Excellent ability with 30% potential, and 70% is required to achieve the above ability (Out Standing, 𝑂𝑆). The assessment results differ from previous studies [8, 9], where the ability level is presented only in linguistic terms, such as Failed, Weak, Normal, etc. This assessment model can present the level of learner ability for each LLO, and the accumulation of all LLOs as the ability level of the course. Based on the learning design in Table 3, this course has 6 LLOs. Table 15 presents the ability level for LLO 2. All ability levels of 𝐿1 are presented in Table 16. For example, 𝐿1 has the ability level (𝐸, 0.1) for LLO 1, (𝐸, −0.1) for LLO 2, (𝑉𝐺, 0.4) for LLO 3, (𝑉𝐺, 0.2) for LLO 4, (𝑉𝐺, 0.4) for LLO 5, and (𝐸, −0.4) for LLO 6. The ability levels LLO 1 to LLO 6 are unified using (19)-(21) to obtain the ability lever for this course, as presented in Table 17. HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 591 Table 15. Ability level results in 2-tuple linguistics Learner Ability Level Description 𝐿1 (𝐸,−0.1) Excellent, although it still takes 10% to reach that ability 𝐿2 (𝐸,−0.2) Excellent, although it still takes 20% to reach that ability 𝐿3 (𝑉𝐺, 0.4) Very Good, there is 40% potential, and 60% mastery is required to achieve the above ability (Excellent, E) 𝐿4 (𝑉𝐺, 0.4) Very Good, there is 40% potential, and 60% mastery is required to achieve the above ability (Excellent, E) 𝐿5 (𝐸, 0.3) Excellent, there is 30% potential, and 70% mastery is required to achieve the above ability (Out Standing, OS) 𝐿6 (𝐸, 0.0) Excellent, 100% is at this level of ability 𝐿7 (𝑉𝐺, 0.2) Very Good, there is 20% potential, and 80% mastery is required to achieve the above ability (Excellent, E) 𝐿8 (𝐺,−0.4) Good, although it still takes 40% to reach that ability 𝐿9 (𝑉𝐺,−0.3) Very Good, although it still takes 30% to reach that ability 𝐿10 (𝐼, 0.4) Insufficient, there is 40% potential, and 60% mastery is required to achieve the above ability (Partially Acceptable, PA) 𝐿11 (𝑉𝐺,−0.5) Very Good, although it still takes 50% to reach this level of ability 𝐿12 (𝑉𝐼, 0.0) Very Insufficient, 100% is at this level of ability 𝐿13 (𝐴,−0.3) Acceptable, although it still takes 30% to reach this level of ability 𝐿14 (𝐸, 0.1) Excellent, there is 10% and required 90% to reach the above level of ability (Out Standing) 𝐿15 (𝐸,−0.5) Excellent, although it still takes 50% to reach this level of ability 𝐿16 (𝐸, 0.0) Excellent, 100% is at this level of ability 𝐿17 (𝑉𝐺, 0.4) Very Good, there is 40% and required 60% to reach the above level of ability (Excellent) 𝐿18 (𝑉𝐺, 0.4) Very Good, there is 40% and required 60% to reach the above level of ability (Excellent) 𝐿19 (𝑉𝐺, 0.4) Very Good, there is 40% and required 60% to reach the above level of ability (Excellent) 𝐿20 (𝑉𝐺,−0.3) Very Good, although it still takes 30% to reach this level of ability Table 16. Ability level results for LLO 1 to LLO 6 in 2-tuple linguistics Learner Ability level (𝒔, 𝜶) 𝑳𝑳𝑶 𝟏 𝑳𝑳𝑶 𝟐 𝑳𝑳𝑶 𝟑 𝑳𝑳𝑶 𝟒 𝑳𝑳𝑶 𝟓 𝑳𝑳𝑶 𝟔 𝐿1 (𝐸, 0.1) (𝐸,−0.1) (𝑉𝐺, 0.4) (𝑉𝐺, 0.2) (𝑉𝐺, 0.4) (𝐸, −0.4) 𝐿2 (𝐸,−0.5) (𝐸,−0.2) (𝐸,−0.5) (𝐸,−0.5) (𝐸,−0.5) (𝑉𝐺, 0.2) 𝐿3 (𝐸,−0.5) (𝑉𝐺, 0.4) (𝑉𝐺, 0.4) (𝑂𝑆, 0.0) (𝐸,−0.1) (𝑉𝐺, 0.4) 𝐿4 (𝐸,−0.1) (𝑉𝐺, 0.4) (𝑉𝐺,−0.1) (𝐸,−0.5) (𝑉𝐺,−0.4) (𝑉𝐺, 0.3) 𝐿5 (𝐺, 0.3) (𝐸, 0.3) (𝑉𝐺, 0.1) (𝐸,−0.4) (𝑉𝐺, 0.1) (𝐸, 0.0) 𝐿6 (𝐺, 0.4) (𝐸, 0.0) (𝑉𝐺, 0.1) (𝐸,−0.5) (𝑉𝐺, 0.1) (𝐸, −0.3) 𝐿7 (𝐸, 0.0) (𝑉𝐺, 0.2) (𝑉𝐺, 0.1) (𝐸,−0.5) (𝑉𝐺, 0.1) (𝐸, −0.4) 𝐿8 (𝐺, 0.3) (𝐺, −0.4) (𝑉𝐺,−0.2) (𝐸,−0.5) (𝑉𝐺, 0.1) (𝐸, 0.1) 𝐿9 (𝐺, 0.4) (𝑉𝐺,−0.3) (𝑉𝐺, 0.4) (𝑉𝐺, 0.3) (𝐺, 0.1) (𝐺, 0.3) 𝐿10 (𝑉𝐺, 0.4) (𝐼, 0.4) (𝑉𝐺, 0.3) (𝑉𝐺, 0.2) (𝐸,−0.1) (𝑉𝐺, 0.4) 𝐿11 (𝐸,−0.2) (𝑉𝐺,−0.5) (𝐴, −0.2) (𝑉𝐺, 0.2) (𝑉𝐺, 0.3) (𝐸, −0.4) 𝐿12 (𝐴, −0.4) (𝑉𝐼, 0.0) (𝐺, 0.3) (𝐼, 0.1) (𝑉𝐼, 0.0) (𝑉𝐼, 0.0) 𝐿13 (𝑉𝐺, 0.2) (𝐴, −0.3) (𝑆, 0.5) (𝐸,−0.5) (𝐺, −0.4) (𝐺, −0.4) 𝐿14 (𝐸,−0.4) (𝐸, 0.1) (𝐸,−0.1) (𝐸,−0.4) (𝑉𝐺, 0.4) (𝐸, −0.3) 𝐿15 (𝑉𝐺,−0.1) (𝐸,−0.5) (𝑆, −0.5) (𝐸,−0.5) (𝑉𝐺, 0.1) (𝐸, −0.4) 𝐿16 (𝐺, 0.3) (𝐸, 0.0) (𝑉𝐺,−0.1) (𝑉𝐺, 0.2) (𝑉𝐺, 0.1) (𝐸, −0.4) 𝐿17 (𝐺, 0.1) (𝑉𝐺, 0.4) (𝑉𝐺,−0.4) (𝐸,−0.5) (𝑉𝐺,−0.3) (𝑉𝐺, 0.0) 𝐿18 (𝑉𝐺, 0.3) (𝑉𝐺, 0.4) (𝑉𝐺, 0.1) (𝐸, 0.3) (𝑆, 0.3) (𝑉𝐺,−0.2) 𝐿19 (𝐸,−0.1) (𝑉𝐺, 0.4) (𝑉𝐺,−0.2) (𝑉𝐺, 0.2) (𝑉𝐺, 0.2) (𝑉𝐺, 0.3) 𝐿20 (𝐸,−0.2) (𝑉𝐺,−0.3) (𝑉𝐺,−0.1) (𝑉𝐺, 0.2) (𝑉𝐺,−0.2) (𝑉𝐺, 0.3) HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 592 Table 17. Ability level results for 6 LLOs in 2-tuple linguistics Learner Ability level (𝒔, 𝜶) Description 𝐿1 (𝐸, −0.4) Excellent, although it still takes 40% to reach that ability 𝐿2 (𝐸, −0.5) Excellent, although it still takes 50% to reach that ability 𝐿3 (𝐸, −0.2) Excellent, although it still takes 20% to reach that ability 𝐿4 (𝑉𝐺, 0.3) Very Good, there is 30% potential, and 70% mastery is required to achieve the above ability (Excellent, E) 𝐿5 (𝑉𝐺, 0.4) Very Good, there is 40% potential, and 60% mastery is required to achieve the above ability (Excellent, E) 𝐿6 (𝑉𝐺, 0.3) Very Good, there is 30% potential, and 70% mastery is required to achieve the above ability (Excellent, E) 𝐿7 (𝑉𝐺, 0.4) Very Good, there is 40% potential, and 60% mastery is required to achieve the above ability (Excellent, E) 𝐿8 (𝑉𝐺,−0.2) Very Good, although it still takes 20% to reach that ability 𝐿9 (𝑉𝐺,−0.2) Very Good, although it still takes 20% to reach that ability 𝐿10 (𝐺, 0.4) Good, there is 40% potential, and 60% mastery is required to achieve the above ability (Very Good, VG) 𝐿11 (𝑉𝐺,−0.5) Very Good, although it still takes 50% to reach that ability 𝐿12 (𝑃𝐴,−0.3) Partially Acceptable, although it still takes 30% to reach that ability 𝐿13 (𝐺, −0.2) Good, although it still takes 20% to reach that ability 𝐿14 (𝐸, −0.3) Excellent, although it still takes 30% to reach that ability 𝐿15 (𝑉𝐺,−0.2) Very Good, although it still takes 20% to reach that ability 𝐿16 (𝑉𝐺, 0.2) Very Good, there is 20% potential, and 80% mastery is required to achieve the above ability (Excellent, E) 𝐿17 (𝑉𝐺,−0.1) Very Good, although it still takes 10% to reach that ability 𝐿18 (𝑉𝐺, 0.0) Very Good, 100% is at this level of ability 𝐿19 (𝑉𝐺, 0.3) Very Good, there is 30% potential, and 70% mastery is required to achieve the above ability (Excellent, E) 𝐿20 (𝑉𝐺, 0.1) Very Good, there is 10% potential, and 90% mastery is required to achieve the above ability (Excellent, E) 4.2.2. Activity Performance Analysis In addition to ability level data, activity performance data is obtained in each LLO stage, including five variables: frequency of attendance (𝑝1), duration of attendance (𝑝2), frequency of access to material (𝑝3), number of posts to the forum (𝑝4), and activity conformance (𝑝5). Table 14 presents a dataset of activity performance values from 20 learners in LLO 2. These performance values are obtained from the event log, which is processed using Figure 3 stages. In Table 14, the performance values of 𝑝1, 𝑝2, 𝑝3, and 𝑝4, are obtained by the code program in Table 9. Then, these performance values are processed into performance levels using Equations 1 and 2 and performance standards Table 8. Changes in performance values 𝑝1, 𝑝2, 𝑝3, 𝑝4, and 𝑝5, into performance levels for learners 𝐿1 to 𝐿20 are presented in Table 19. In Table 14, the performance value of the variable 𝑝5 is obtained with the program code Table 9. The fitness value indicates the conformance of the learner’s actual activity against the instructed activity design in the range [0,1]. This fitness value is obtained based on Figure 3. The process of event logs data begins with extraction, then preprocessing and formatting. Calculating the fitness value is carried out with an alignment-based algorithm, which begins with creating a process model from the activity design, as shown in Figure 4, and the process model of the actual learner’s activities, as shown in Figure 6 and Figure 7. Then, a reply trace of the activity between the two process models is carried out, as presented in Figure 8 for 𝐿3, and Figure 9 for 𝐿10. From the replies between these traces, the cost values, the sum of the event activities, and the shortest path through the Petri Net are obtained, and the fitness values are obtained using (11). The fitness measurement process uses the program code in Table 9 using Equation 12, as presented in Table 18. For example, according to Figure 8, 𝐿3 has a cost value 3 and a bwc value 17. The fitness value calculated by Equation 12 is 0.824. Table 14 shows that each learner has five activity performance variables, namely frequency of attendance (𝑝1), duration of attendance (𝑝2), frequency of access material (𝑝3), number of posts to forum (𝑝4), and activity conformance (𝑝5). The 𝑝5 value in Table 14 is obtained from the cost and bwc from event logs, which are calculated using Equation 12, as presented in Table 18. For example, for 𝐿1, the fitness value of 0.667 is obtained from a cost value of 5 and bwc 15. The analysis of activity performance in this model presents results in the form of performance levels from the five activity performance variables for each learner. Therefore, the activity performance data in Table 14 is then converted into performance levels based on the performance standards provisions of Table 8. For example, for 𝐿1, with a value of 𝑝1is 18, 𝑝2 is 869, 𝑝3 is 1, 𝑝4 is 1, and 𝑝5 is 0.667, the levels for the five activity performance variables are High, High, Low, Medium, and Medium, as presented in Table 19. HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 593 Figure 6. Process models 𝑳𝟑 with fitness values 0.824. Figure 7. Process models 𝑳𝟏𝟎 with fitness values 0.462 Figure 8. Mapping alignment 𝑳𝟑 Figure 9. Mapping alignment 𝑳𝟏𝟎 Table 18. Fitness value for activity conformance Learner Cost bwc Fitness value 𝐿1 5 15 0.667 𝐿2 6 14 0.571 𝐿3 3 17 0.824 𝐿4 4 18 0.778 𝐿5 3 17 0.824 𝐿6 3 17 0.824 𝐿7 7 13 0.462 𝐿8 7 19 0.632 𝐿9 6 14 0.571 𝐿10 7 13 0.462 𝐿11 6 16 0.625 𝐿12 0 0 0.000 𝐿13 6 14 0.571 𝐿14 5 15 0.667 𝐿15 6 16 0.625 𝐿16 6 14 0.571 𝐿17 6 14 0.571 𝐿18 4 16 0.750 𝐿19 5 15 0.667 𝐿20 6 16 0.625 HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 594 Table 19. Data variable activities performance Learner 𝒑𝟏 𝒑𝟐 𝒑𝟑 𝒑𝟒 𝒑𝟓 𝐿1 High High Low Medium Medium 𝐿2 High High Medium Low Medium 𝐿3 Low Medium Low Low High 𝐿4 Low Medium Low Low High 𝐿5 High High High Medium High 𝐿6 High Low Medium Low High 𝐿7 Medium Low Low Low Medium 𝐿8 High High Low Low Medium 𝐿9 High High Medium Low Medium 𝐿10 Medium High Low Low Medium 𝐿11 Medium High Low Low Medium 𝐿12 Low Low Low Low Low 𝐿13 Medium High Low Low Medium 𝐿14 Low Low Low Low Medium 𝐿15 Medium High Medium Low Medium 𝐿16 Medium Medium Low Low Medium 𝐿17 Low Low Low Low Medium 𝐿18 Medium Medium Low Low High 𝐿19 Low High Low Low Medium 𝐿20 Medium High Low Low Medium 4.2.3. Recommendation Based on Table 12 and Table 15, the recommendations given are in the form of recommendations for LLO stages and activity performance directives. Table 20 presents examples of recommendations for 𝐿1 and 𝐿10. From Table 15, the 𝐿1 ability level is (E, -0.1). Because the value 𝑠 is E, 𝐿1 is Succeed and can proceed to LLO 3. Table 20. Recommendation to the learner Learner LLO stages Activity performance directions 𝐿1 Succeed. Continue to LLO 3 • High frequency of attendance. Keep it up! • High duration of attendance. Keep it up! • Low material access. Open and read the material provided so you can understand the material presented! • Participation in forums is medium. Increase involvement in forums! • Compatibility of activities with teacher instruction is Medium. Improve obedience to the teacher’s guides for even better learning outcomes! 𝐿10 Failed. You must repeat LLO 2 • Medium attendance. Increase attendance so that learning outcomes are better. • High duration of attendance. Keep it up! • Low material access. Open and read the material provided so you can understand the material presented! • Participation in forums is Low. Join the discussion and share your opinion! • The compatibility of activities with teacher instruction is Medium. Improve adherence to instructions for even better learning outcomes! Meanwhile, the 𝐿10 ability level is (I, 0.1). Because the value 𝑠 is I, 𝐿10 Failed and must improve the LLO 2 attainment. Apart from recommendations related to LLO, both 𝐿1 and 𝐿10 receive directions for improvements to enhance future performance. The recommendation is different from previous studies [6, 48, 49], which focused on recommendations related to material learning 4.3. Discussion In this study, e-learning is constructed with OBE as a pedagogical-based methodology. The course learning design in Table 3 presents a way for OBE to maintain alignment of learning outcomes, processes, and assessments [10]. The learning design table explains that this course has several LLOs, with 3 LLOs as examples. Each LLO is associated with HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 595 one or more CLOs, has indicators to measure its attainment, and has a duration of time. The learning activities column explains that LLO is implemented through a process in the form of a complete and structured series of learning activities [2, 39], as presented in Table 4. This entire series of activities includes presenting video materials and PDF documents, discussions, and assignments. This variety of activities aims to encourage learner involvement in e-learning. In addition to being complete, the series of activities are arranged in a structured and sequential manner to guide learners to attain learning outcomes. The attainment of LLO is measured through several techniques and integrative assessment instruments [2]. From this description, the learning design shown in this study differs from the implementation of OBE in e-learning by Pusparini (2020) [39], which did not formulate the alignment of learning outcomes, processes, and assessments. The learning design of this model is presented in the LMS e-learning as in Figures 10 and 11. Figure 10. View of CLOs and LLOs on the instructor dashboard Figure 11. View of learning design on the instructor dashboard HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 596 In online learning situations, the presence and involvement of learners affect performance and learning outcomes [1]. This study formulates five activity performance variables that are relevant to support the assessment, namely frequency of attendance (𝑝1), duration of attendance (𝑝2), frequency of access material (𝑝3), number of posts to forum (𝑝4), and activity conformance (𝑝5), such as presented in Table 5. The performance dataset was obtained from the event logs, analyzed based on Figure 3, and the program code Table 9. The results of calculating the five activity performance variables are numerical values, as presented in Table 14. Calculating the value of fitness activity conformance (𝑝5) is done by alignment based on the code program Table 9. Based on the alignment-based analysis, each learner has several traces from each login-logout session. A process model is created with Petri Net at each trace, as shown in Figure 6 and Figure 7. This process model is compared with the activity design process model, as shown in Figure 4, using Equation 11. Each learner will have several fitness values according to the number of traces. The highest fitness value is taken from a number of these, representing the conformance of the learner’s best activity against the activity design provided by the teacher, as presented in Table 18. The activity design provided by the teacher becomes a reference for activities so that students are guided in achieving learning outcomes. It differs from the previous study [22], which used the best learner activities as a reference. The numerical activity performance values in Table 14 are further processed with Equations 1 and 2, the performance standards in Table 8, and assigned as performance levels in High, Medium, or Low, as presented in Table 19 and Figure 12. Figure 12. View activity performance data and the level of performance on the instructor dashboard The dataset from the event logs is not only used for activity performance analysis. The event logs data also analyze disciplinary attitudes by calculating the lateness in submitting assignments. Calculations are performed based on activity and timestamp attributes. The lateness is converted into a disciplinary attitude value using Table 10. For example, if a learner is late in submitting a quiz and assignment in the 1 − 6 hours range, then the value of the attitude of discipline is 75. The value of this attitude of discipline becomes part of the dataset for determining the ability level. The ability level represents the result of a comprehensive assessment, including elements of knowledge, skills, and attitudes, as formulated in CLO Table I. Therefore, various instruments are used in this assessment, namely quizzes, e- rubrics, and log recordings. This study adds an e-rubric to the LMS to overcome the limitations of e-learning in providing assessment instruments as needed [14]. The addition of e-rubric in this LMS is in line with OBE which recommends utilizing rubrics in a comprehensive assessment framework [2]. The quiz instrument offers an element of knowledge assessment data in numerical form. E-rubric provides an element of skill assessment data in linguistic form. It differs from previous studies [28, 30], which provided assessment data in numerical form. The existence of dimensions in e-rubric, as presented in Figure 2, facilitates the assessment of complex and detailed skill elements. In addition, the e-rubric can accommodate assessment data in linguistics. This ability is added value because, according to Ho et al. (2020) [32], compared to the numerical, description of skill dimension in linguistics, it enables learners to understand their current conditions, and what attainment should be expected. Unfortunately, the variety of numerical and linguistic assessment data causes problems in the merging process [33, 34]. This study proposes using the 2-tuple fuzzy linguistic method to overcome the problem of merging. This method can avoid the loss of important information in the assessment caused by various forms of data [11]. The 2-tuple fuzzy linguistic in this assessment system is carried out in two stages. First, aggregate linguistic values from the four e-rubric dimensions, as presented in (23)-(25). Second, the unification of the three data assessments, namely the quiz scores in numeric [0-100], the results of aggregating the e-rubric dimensions in (𝑠, 𝛼), and the discipline value from logs recording in numeric [0- 100]. This unification produces a level of ability in the form (𝑠, 𝛼), as presented in Table 15. The form (𝑠, 𝛼) can differentiate the level of the learner’s ability more specifically, where 𝑠 indicates the level of ability attained by the learner HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 597 in linguistics, while 𝛼 is a numerical value that distinguishes the ability of the learner from other learners in the same linguistic and indicates the potential to achieve higher learning outcomes. For example, in Table 15, the ability level of 𝐿1 is (𝐸, −0.1) and 𝐿5 is (𝐸, 0.3). Even though 𝐿1 and 𝐿5 have the same ability level. They can be explicitly distinguished because they have different 𝛼 values. The various assessment data and ability levels in a 2-tuple linguistic form are presented in Figure 13. In addition to unifying the e-rubric dimension and the variety of assessment results per LLO, the 2- tuple fuzzy linguistic method can aggregate the ability levels of all LLOs in this course, from LLO 1 to LLO 6 using (19)- (21). The aggregation result will be the learner’s ability level in the course, as presented in Table 17 and Figure 14. Presenting the assessment results on each learning stage was not carried out by Andayani (2017) [11], while aggregation of all stages was not presented by Umer et al. (2017) [22]. Figure 13. Learner ability level for LLO 2 Figure 14. Recapitulation of learning outcomes for all LLOs The assessment process for each LLO produces the level of ability and performance of the learner’s activities. Equation 22 will process the 𝑠 values of the ability level and performance levels into recommendations. The value of 𝑠 will determine whether the learner Succeed or Failed in LLO. The level of performance in each variable will determine the performance direction given, as presented in Table 19. For example, according to Table15, 𝐿1 with the ability level (𝐸, −0.1) is declared Succeed. Based on the performance level of Table 19, 𝐿1 has a performance level of High for attendance frequency (𝑝1), High for duration attendance (𝑝2), Low for frequency of material access (𝑝3), Medium for number of posts to forum (𝑝4), and Medium for activity conformance (𝑝5), then 𝐿1 gets directions as presented in Tables 19 and 20. Because the attendance frequency is high, learners are advised to keep this performance. For attendance duration is high, learners are directed to keep this duration. If the frequency of material access is low, learners are required to open and read materials, both videos and PDF documents, to understand the material delivered. For the medium number of posts to the forum, learners are directed to increase participation in the discussion. Finally, for medium-activity conformance, learners are directed to improve obedience to the teacher’s guides for even better learning outcomes. Recommendations regarding success or failure for each LLO and activity performance directions are presented on the HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 598 learner dashboard, as shown in Figures 15 and 16. The information on Succeed is presented in a green box, while the Failed information is in a red box. Activity performance directions are presented in gray boxes. Figure 15. Ability level and recommendation for Succeed learners on the learner dashboard Figure 16. Ability level and recommendation for Failed learners on the learner dashboard This assessment model generally differs from previous research assessment models [39]. Even though previous research has implemented OBE, Pusparini (2020) [39] did not add an e-rubric to meet the needs of assessing complex and detailed aspects. It did not manage various combined numerical and linguistic assessment data and does not utilize 2-tuple fuzzy linguistics as a model for representing assessment results. There are several differences between using the 2-tuple fuzzy linguistic representation model in this research and Andayani (2017) [11]. First, the cardinality of the linguistic terms used where Andayani (2017) [11] uses seven cardinalities while this model uses nine cardinalities. The second difference lies in determining Triangular Fuzzy Notation (TFN) semantics, Andayani (2017) [11] uses a symmetric approach with a mean value of 0.5. In contrast, the TFN semantics in this assessment model uses an asymmetric approach, as explained in previous studies [33, 34]. This model also accommodates the actual conditions of the value interval provisions that apply to universities that are the object of research. The 2-tuple fuzzy linguistic representation model presents the ability level in a 2-tuple form. It differs from previous research, where the ability level was only in linguistic terms [8, 9]. The final value in 2-tuple form (𝑠, 𝛼) is more meaningful. Learning experiences in the form of learning activities are stored naturally in the e-learning LMS. This record shows the performance of learner activities that can be analyzed. This research examines this performance based on the object's frequency, time (timestamp), and sequence of activities. The analysis is formulated in five relevant performance variables, namely frequency of attendance (𝑝1), duration of attendance (𝑝2), frequency of accessing material (𝑝3), number of opinions in the forum (𝑝4), and conformity of learner activities with normative activity design (𝑝5). The recommendations for learning stages and activity performance directions in this model are different from previous research [6, 48, 49], which focused on recommendations related to learning materials and learning paths. This model provides recommendations for learning stages and directions for activity performance. 4.4. Model Performance Measurement Model performance measurements are carried out to ensure the quality of this assessment model. Performance measurement uses Standard Error of Measurement (SEM) and a model acceptance questionnaire by users. SEM is used to compare assessment scores without 2-tuple fuzzy linguistics and scores using 2-tuple fuzzy linguistics. Table 21 presents SEM results for assessments using and not using 2-tuple fuzzy linguistics. Table 21 presents the SEM results for each LLO and all LLOs (CLO). HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 599 Table 21. Performance comparison Learning outcome Standard error of measurement without 2-tuple fuzzy linguistic with 2-tuple fuzzy linguistic 𝐿𝐿𝑂1 4.372 5.778 𝐿𝐿𝑂2 3.481 4.383 𝐿𝐿𝑂3 5.183 5.157 𝐿𝐿𝑂4 3.404 3.352 𝐿𝐿𝑂5 4.318 4.483 𝐿𝐿𝑂6 3.104 2.820 𝐶𝐿𝑂 3.853 3.385 Based on Table 21, it is known that the standard error value, as an indication of the distribution of measurement error to estimate the actual and obtained student scores, for assessments using the 2-tuple fuzzy linguistics representation model is 3.385. This value is smaller than the standard error value for assessments without a 2-tuple fuzzy linguistics representation model of 3.853. The smaller the error, the better the assessment. Apart from SEM, performance measurement is also carried out using a questionnaire that was developed independently and validated by experts to find out several important parts that can improve the e-learning assessment model. The selected individuals or respondents were 5 teaching lecturers and group lecturers. Furthermore, the questionnaire results show that user acceptance of the model was 3.617 or 90.43%, meaning it was very well received. 5. Conclusions In the context of implementing online learning properly and relevant to the needs of the current learning paradigms, this study proposed a learner assessment system in e-learning with the following capabilities: • E-learning is constructed with OBE. The learning design describes how OBE is formulated to maintain alignment between learning outcomes (CLOs, LLOs, and LLO indicators), process, and assessment. • This assessment system utilizes event logs to analyze learner activity performance and realize comprehensive assessment needs. Through this performance analysis, learners’ disciplinary attitudes represent attitude assessment elements. Apart from discipline, this analysis can also explore five performance activity variables as input in providing recommendations: frequency of attendance, duration of attendance, frequency of access material, number of posts to the forum, and activity conformance. • This assessment system can unify numerical and linguistic assessment data from various assessment instruments, including e-rubric added to LMS e-learning, and present the unification results as ability levels in the form of 2-tuple (𝑠, 𝛼) to realize comprehensive assessment needs. • Using a rule base, this assessment system processes the ability levels and activity performance into recommendations regarding LLO attainment and performance directions. Academic stakeholders are expected to utilize reports on learning outcomes from this system according to their respective needs. This assessment system can be applied to universities and online learning providers as a form of progress in e- learning and to adequately meet the needs of online learning. The system is limited to two conditions. First, in determining the Succeed or Failed status of an LLO stage, the minimum requirement for the ability level to have Succeed status is the linguistic term Acceptable or a numerical value of 3. A learner with a numerical value below 3, for example, 2.88, will have Succeed status because rounding up the numerical value is 3 in the form of 2-tuple (𝐴, −0.12), which is Acceptable, although it still takes 12% to reach that ability. Second, the weight of the assessment items affects the ability level calculation results. For example, in 𝐿10, even though the learner takes the quiz and submits the assignment, the status will be Failed if both scores are bad. Meanwhile, learners who do not take the quiz but have a high assignment score will be successful, for example, in 𝐿13. This assessment model can be used in future studies and various other courses. The assessment technique and instruments in each LLO can be more varied to aggregate more assessment items with 2-tuple fuzzy linguistics. E-rubric can also be used to accommodate numerical assessment data. The use of a decision support system to calculate the weight of assessment items can be considered with the aim of weighting assessment items more objectively so that the assessment results are fairer. In terms of event logs, event logs analysis can be expanded to different perspectives or other process mining techniques, such as enhancement, while still being based on the science of assessment in education as the foundation. HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 600 6. Declarations 6.1. Author Contributions Conceptualization, W.D.Y. and S.H.; methodology, W.D.Y., S.H., S.P., and H.D.S.; software, W.D.Y., S.P., and H.D.S.; validation, W.D.Y. and H.D.S.; formal analysis, S.H. and S.P.; resources, W.D.Y. and S.H.; data curation, W.D.Y. and S.P.; writing—original draft preparation, W.D.Y. and S.H.; writing—review and editing, W.D.Y., S.H., S.P., and H.D.S.; supervision, S.H., S.P., and H.D.S.; funding acquisition, S.H. All authors have read and agreed to the published version of the manuscript. 6.2. Data Availability Statement The data presented in this study are available on request from the corresponding author. 6.3. Funding This work was supported by The Research Directorate of Universitas Gadjah Mada, Yogyakarta, Indonesia, in the RTA program Universitas Gadjah Mada with Grant Number 5075/UN1.P.II/Dit-Lit/PT.01.01/2023. 6.4. Acknowledgements The authors would like to thank the Universitas Islam Negeri Walisongo Semarang for providing data for this study. 6.5. Institutional Review Board Statement Not applicable. 6.6. Informed Consent Statement Written informed consent from the participants was not required to participate in this study in accordance with the national legislation and the institutional requirements. 6.7. Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. 7. References [1] Dwi Yuniarti, W., Winarko, E., & Musdholifah, A. (2020). Data mining for student assessment in e-leaming: A survey. 2020 5th International Conference on Informatics and Computing, ICIC 2020, 348–354. doi:10.1109/ICIC50835.2020.9288533. [2] Kemendikbud, D. J. P. T. (2020). Guidelines for compiling the higher education curriculum in the industrial era 4.0 to support the independent campus learning program. The Directorate General of Higher Education, Ministry of Education and Culture Republik of Indonesia, Jakarta, Indonesia. [3] Salybekova, N., Abdimalik, S., Issayev, G., Khalikova, G., Berdenkulova, A., & Bakirova, K. (2023). E-Learning Adoption: Designing a Network-Based Educational and Methodological Course on “Humans and Their Health.” Emerging Science Journal, 7(6), 2097–2119. doi:10.28991/ESJ-2023-07-06-014. [4] Cerezo, R., Bogarín, A., Esteban, M., & Romero, C. (2020). Process mining for self-regulated learning assessment in e-learning. Journal of Computing in Higher Education, 32(1), 74–88. doi:10.1007/s12528-019-09225-y. [5] Bezhovski, Z., & Poorani, S. (2016). The Evolution of E-Learning and New Trends. Information and Knowledge Management, 6(3), 50–57. [6] Budiyanto, U., Hartati, S., & Azhari. (2018). e-Learning Model According to the Learning Style and Ability level of Students. Universitas Gadjah Mada, Yogyakarta, Indonesia. [7] Bryson, C., & Hand, L. (2007). The role of engagement in inspiring teaching and learning. Innovations in Education and Teaching International, 44(4), 349-362. doi:10.1080/14703290701602748. [8] Azimjonov, J. (2016). Evaluation of Distance Learning Students Performance Using Fuzzy Logic. Journal of Management Information Systems, 1(October), 87–97. [9] Wardoyo, R., & Yuniarti, W. D. (2020). Analysis of Fuzzy Logic Modification for Student Assessment in e-Learning. International Journal on Informatics for Development, 9(1), 29. doi:10.14421/ijid.2020.09105. [10] Qadir, J., Shafi, A., Al-Fuqaha, A., Taha, A. E. M., Yau, K. L. A., Ponciano, J., Hussain, S., Ali Imran, M., Muhammad, S. S., Rais, R. N. Bin, Rashid, M., & Tan, B. L. (2020). Outcome-based (engineering) education (OBE): International accreditation practices. ASEE Annual Conference and Exposition, Conference Proceedings, 2020-June, 28865. doi:10.18260/1-2--35020. HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 601 [11] Andayani, S. (2017). ALiDT model for a decision support system based on fuzzy multiple criteria decision making in determining student learning outcomes profiles. Ph.D. dissertation, Department Computer and Electronics, Universitas Gadjah Mada, Yogyakarta, Indonesia. [12] Agustianto, K., Permanasari, A. E., Kusumawardani, S. S., & Hidayah, I. (2016). Design adaptive learning system using metacognitive strategy path for learning in classroom and intelligent tutoring systems. AIP Conference Proceedings, 1755, 70012. doi:10.1063/1.4958507. [13] Saito, T., & Watanobe, Y. (2020). Learning path recommendation system for programming education based on neural networks. International Journal of Distance Education Technologies, 18(1), 36–64. doi:10.4018/IJDET.2020010103. [14] Lara, J. A., Aljawarneh, S., & Pamplona, S. (2020). Special issue on the current trends in E-learning Assessment. Journal of Computing in Higher Education, 32(1), 1–8. doi:10.1007/s12528-019-09235-w. [15] Jacob, D., & Henriques, R. (2023). Educational Data Mining to Predict Bachelors Students’ Success. Emerging Science Journal, 7(Special Issue 2), 159–171. doi:10.28991/ESJ-2023-SIED2-013. [16] Cerezo, R., Esteban, M., Sánchez-Santillán, M., & Núñez, J. C. (2017). Procrastinating behavior in computer-based learning environments to predict performance: A case study in Moodle. Frontiers in Psychology, 8, 1403. doi:10.3389/fpsyg.2017.01403. [17] Opre, D., Șerban, C., Veșcan, A., & Iucu, R. (2024). Supporting students’ active learning with a computer based tool. Active Learning in Higher Education, 25(1), 135-150. doi:10.1177/14697874221100465. [18] Al-Shabandar, R., Hussain, A. J., Liatsis, P., & Keight, R. (2019). Detecting at-risk students with early interventions using machine learning techniques. IEEE Access, 7, 149464–149478. doi:10.1109/ACCESS.2019.2943351. [19] Waheed, H., Hassan, S. U., Aljohani, N. R., Hardman, J., Alelyani, S., & Nawaz, R. (2020). Predicting academic performance of students from VLE big data using deep learning models. Computers in Human Behavior, 104. doi:10.1016/j.chb.2019.106189. [20] Haiyang, L., Wang, Z., Benachour, P., & Tubman, P. (2018). A time series classification method for behaviour-based dropout prediction. Proceedings - IEEE 18th International Conference on Advanced Learning Technologies, ICALT 2018, 191–195. doi:10.1109/ICALT.2018.00052. [21] Hussain, M., Zhu, W., Zhang, W., & Abidi, S. M. R. (2018). Student Engagement Predictions in an e-Learning System and Their Impact on Student Course Assessment Scores. Computational Intelligence and Neuroscience, 6347186. doi:10.1155/2018/6347186. [22] Umer, R., Susnjak, T., Mathrani, A., & Suriadi, S. (2017). On predicting academic performance with process mining in learning analytics. Journal of Research in Innovative Teaching & Learning, 10(2), 160–176. doi:10.1108/jrit-09-2017-0022. [23] Okubo, F., Shimada, A., Yamashita, T., & Ogata, H. (2017). A neural network approach for students’ performance prediction. ACM International Conference Proceeding Series, 598–599. doi:10.1145/3027385.3029479. [24] Yang, F., & Li, F. W. B. (2018). Study on student performance estimation, student progress analysis, and student potential prediction based on data mining. Computers and Education, 123(October), 97–108. doi:10.1016/j.compedu.2018.04.006. [25] Azizah, E. N., Pujianto, U., Nugraha, E., & Darusalam. (2018). Comparative performance between C4.5 and Naive Bayes classifiers in predicting student academic performance in a Virtual Learning Environment. 2018 4th International Conference on Education and Technology, ICET 2018, 1, 18–22. doi:10.1109/ICEAT.2018.8693928. [26] Riz, G., Santos, E. A. P., & de Freitas Rocha Loures, E. (2017). Interoperability assessment in health systems based on process mining and MCDA methods. Advances in Intelligent Systems and Computing, 569, 436–445. doi:10.1007/978-3-319-56535- 4_44. [27] Van der Aalst, W., Adriansyah, A., & Van Dongen, B. (2012). Replaying history on process models for conformance checking and performance analysis. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 2(2), 182–192. doi:10.1002/widm.1045. [28] Ana, A., Yulia, C., Jubaedah, Y., Muktiarni, M., Dwiyanti, V., & Maosul, A. (2020). Assessment of student competence using electronic rubric. Journal of Engineering Science and Technology, 15(6), 3559–3570. [29] Ibáñez, J. C., Saiz, M. S. I., & Gómez, G. R. (2014). A proposal for skill evaluation via complex tasks in virtual learning environments. ACM International Conference Proceeding Series, 429–433. doi:10.1145/2669711.2669935. [30] Muktiarni. (2019). Android-based e-rubric to measure food service competence in vocational education. M.S. Thesis, Master of Vocational Technology Education, Universitas Pendidikan Indonesia, Bandung, Indonesia. [31] Brookhart, S. M. (2018). Appropriate Criteria: Key to Effective Rubrics. Frontiers in Education, 3(April), 22. doi:10.3389/feduc.2018.00022. [32] Ho, Y. Y., & Lim, W. Y. R. (2020). Educating Adult Learners: Bridging Learners’ Characteristics and the Learning Sciences. Diversity and Inclusion in Global Higher Education: Lessons from Across Asia, 97–115. doi:10.1007/978-981-15-1628-3_4. HighTech and Innovation Journal Vol. 5, No. 3, September, 2024 602 [33] Herrera, F., & Martinez, L. (2000). An approach for combining linguistic and numerical information based on the 2-tuple fuzzy linguistic representation model in decision-making. International Journal of Uncertainty, Fuzziness and Knowldege-Based Systems, 8(5), 539–562. doi:10.1142/S0218488500000381. [34] Herrera, F., & Martinez, L. (1996). Brief Papers. Brain and Cognition, 32(2), 273–344. doi:10.1006/brcg.1996.0066. [35] Amelia, N., Abdullah, A. G., & Mulyadi, Y. (2019). Meta-analysis of student performance assessment using fuzzy logic. Indonesian Journal of Science and Technology, 4(1), 74–88. doi:10.17509/ijost.v4i1.15804. [36] Sudaryono, Rahardja, U., & Masaeni. (2020). Decision Support System for Ranking of Students in Learning Management System (LMS) Activities using Analytical Hierarchy Process (AHP) Method. Journal of Physics: Conference Series, 1477(2), 1–7. doi:10.1088/1742-6596/1477/2/022022. [37] Hasibuan, M. S., Abdul Aziz, R. Z., Dewi, D. A., Kurniawan, T. B., & Syafira, N. A. (2023). Recommendation Model for Learning Material Using the Felder Silverman Learning Style Approach. HighTech and Innovation Journal, 4(4), 811–820. doi:10.28991/HIJ-2023-04-04-010. [38] Klifartha, N. (2008). Redesign of the Assessment Module on the I-Elisa e-Learning System. Universitas Gadjah Mada, Yogyakarta, Indonesia. [39] Pusparini, M. D. (2020). Outcome Based Education (OBE) Online Learning with Molta. Refleksi Pembelajaran Inovatif, 2(2), 336–357. doi:10.20885/rpi.vol2.iss2.art5. [40] Mahbubul Syeed, M. M., Shihavuddin, A. S. M., Uddin, M. F., Hasan, M., & Khan, R. H. (2022). Outcome Based Education (OBE): Defining the Process and Practice for Engineering Education. IEEE Access, 10(9), 119170–119192. doi:10.1109/ACCESS.2022.3219477. [41] Yanes, N., Mostafa, A. M., Ezz, M., & Almuayqil, S. N. (2020). A machine learning-based recommender system for improving students learning experiences. IEEE Access, 8(ii), 201218–201235. doi:10.1109/ACCESS.2020.3036336. [42] Verma, S. K., Thakur, R. S., & Jaloree, S. (2017). Fuzzy association rule mining-based model to predict students’ performance. International Journal of Electrical and Computer Engineering, 7(4), 2223–2231. doi:10.11591/ijece.v7i4.pp2223-2231. [43] Casas-Ramos, J., Mucientes, M., & Lama, M. (2024). REACH: Researching Efficient Alignment-based Conformance Checking. Expert Systems with Applications, 241(January 2023), 122467. doi:10.1016/j.eswa.2023.122467. [44] Park, G., Adams, J. N., & van der Aalst, W. M. P. (2022). OPerA: Object-Centric Performance Analysis. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 13607 LNCS, 281–292. doi:10.1007/978-3-031-17995-2_20. [45] Jokonowo, B., Siti Fatonah, N., & Akhir, E. A. P. (2021). Conformance Checking of Dwelling Time Using a Token-based Method. Journal of Information Systems Engineering and Business Intelligence, 7(2), 129. doi:10.20473/jisebi.7.2.129-137. [46] Fnukal, K. (2021). Implementation of advanced process mining algorithms in .NET. Masaryk University, Brno, Czechia. [47] Adeyemo, S. a. (2010). Students’ Ability Level and Their Competence in Problem-Solving Task in Physics. International Journal of Educational Research and Technology, 1(December), 35–47. [48] Herath, D., & Jayarathne, L. (2018). An Architecture for a Personalized Learning Recommendation on Knowledge Levels of Learner. International Journal of Scientific & Engineering Research, 9(6). 98-103. [49] Wang, X., Gulenman, T., Pinkwart, N., De Witt, C., Gloerfeld, C., & Wrede, S. (2020). Automatic assessment of student homework and personalized recommendation. Proceedings - IEEE 20th International Conference on Advanced Learning Technologies, ICALT 2020, 150–154. doi:10.1109/ICALT49669.2020.00051. [50] Yuniarti, W. D., Hartati, S., & Priyanta, S. (2022). a Preliminary Learner Assessment Framework on E-Learning. ICIC Express Letters, 16(7), 705–712. doi:10.24507/icicel.16.07.705. [51] Yuniarti, W. D., Hartati, S., Priyanta, S., & Surjono, H. D. (2023). Data Unification Utilizing E-Rubric for Learner Assessment on E-Learning with OBE Approach. ICIC Express Letters, 17(8), 927–935. doi:10.24507/icicel.17.08.927. [52] Goghari, V. M., & Landrum, R. E. (2021). Teaching psychology (Editorial). Canadian Psychology, 62(4), 345–347. doi:10.1037/cap0000310. [53] Widhiarso, W. (214). Categorizing data using hyphothetical statistics and empirical statistics. Faculty of Psychology, Universitas Gajah Mada, Yogyakarta, Indonesia. [54] Yuniarti, W. D., Hartati, S., Priyanta, S., & Surjono, H. D. (2022). Utilization of Linguistic Data for Learner Assessment on e- Learning: Instrument and Processing. 2022 7th International Conference on Informatics and Computing, ICIC 2022, 329–333. doi:10.1109/ICIC56845.2022.10006977.