BCP Social Sciences & Humanities ERSS 2024 Volume 23 (2024) 65 Research on the Application of Cognitive Diagnostic Models in Assessment Evaluation Juan Peng a, Mengxue Xu b School of Xi'an Eurasia University, Xi’an Shaanxi, 710065, China a pengjuan@eurasia.edu, b xumengxue@eurasia.edu Abstract. With the integration of cognitive diagnostic technology into the field of education, meeting individual learning needs has become possible. This study takes the “Excel Data Analysis” module in the “University Computer Basics” course as the research subject, aiming to evaluate the application effect of Cognitive Diagnostic Models in student assessment and evaluation. Through in- depth interviews with teachers and experts, the key competencies that students need to master were clarified, including data understanding and basic operations, data computation and processing, data management and analysis, data visualization, etc. This study constructed a DINA model, established a hierarchical structure of competence attributes, defined the Q-matrix and the ideal competence mastery pattern, and used the flexCDMs platform for parameter estimation, analyzing the test item parameters and the probability of students' competence attributes mastery. The results show that students perform well in “data computation and processing,” while “advanced analysis skills” need to be strengthened. The study provides evidence for teaching improvement, and although there are limitations, it points out the direction for future research. Keywords: Cognitive Diagnostic Models; Excel Data Analysis; DINA Model; Assessment Evaluation. 1. Introduction In the field of education, assessment is not only an important means to measure the effectiveness of students' learning, but also a key link in guiding the improvement of teaching and promoting students' development. With the deepening of education reform, the traditional score-oriented assessment can no longer meet the needs of individualized learning and comprehensive development. Therefore, it is particularly important to find an evaluation method that can reflect students' cognitive ability and knowledge mastery more comprehensively and accurately. Cognitive A3agnostic Model (CDM), as an emerging educational assessment tool, reveals the cognitive ability and knowledge structure behind students by analyzing their performance on specific cognitive tasks. This model focuses not only on the final performance of students, but also on the cognitive characteristics and developmental potential of students during the learning process [1]. “Excel Data Analysis”, as a core module in the ‘University Computer Fundamentals’ course, is crucial to the development of students' data-processing ability, logical thinking ability, and ability to solve practical problems. Traditional assessment methods often focus on students' operational proficiency and task completion, but it is difficult to assess students' cognitive structure and ability development in depth. This leads to incomplete and inaccurate assessment and evaluation, and fails to provide effective feedback for teaching improvement and students' individualized development. Cognitive diagnostic theory emphasizes the comprehensive assessment of students' ability development, which helps to understand students' learning status more accurately and provides a scientific basis for teaching improvement. This study aims to explore the applicability of the Cognitive Diagnostic Models in the assessment and evaluation of the module “Excel Data Analysis”, and how to provide more accurate assessment of students' cognitive ability and knowledge mastery through the model, so as to provide support for teaching practice and students' personalized learning. BCP Social Sciences & Humanities ERSS 2024 Volume 23 (2024) 66 2. Cognitive Diagnostic Theory 2.1 Theory of Cognitive Diagnostic Assessment Cognitive diagnostic assessment is based on the research results of modern measurement and cognitive science, using modern statistical methods and computer technology as tools to deeply analyze students' cognitive activities and their development in the process of answering questions by employing fine mathematical models. The core idea of the cognitive diagnostic assessment theory is to combine cognitive processes with measurement tools to not only assess the overall ability level of an individual, but also further diagnose his/her cognitive structure and individual differences. The theory emphasizes an in-depth analysis of the knowledge and cognitive processing skills involved in solving a particular task, and reveals the potential cognitive strengths and deficiencies of an individual by constructing a cognitive model [1]. The research and development of Cognitive Diagnostic Models is one of the centers of the theory, which has been in development for more than 50 years. According to statistics, there have been more than 100 Cognitive Diagnostic Models, which have their own characteristics and scope of application, but all of them are committed to a deeper understanding and improvement of students' cognitive processes and knowledge structures. In this paper, the DINA model is selected as a research object, aiming to investigate the effectiveness of Cognitive Diagnostic Models in the evaluation of students in basic computer courses. 2.2 DINA Cognitive Diagnostic Models The DINA (Deterministic Inputs, Noisy “And” gate model) model is an influential Cognitive Diagnostic Models in the field of modern psychological and educational measurement. The DINA model was developed based on item response theory. It assumes that a subject's correct response to an item depends on whether or not he or she has mastered the specific knowledge points (attributes) corresponding to that item. If the subject has mastered all the relevant attributes, then ideally he or she should be able to answer the item correctly; however, the actual situation may deviate due to factors such as measurement error. The item response function of the DINA model is: (1) included among these (2) Specifically, there are two key parameters in the DINA model: the slipping parameter and the guessing parameter. The slipping parameter, ,indicates the probability that the subject has mastered all the attributes of the item measurement but still answers the item incorrectly. The guessing parameter, ,indicates the probability that a subject can answer the item correctly even though he/she does not have mastery of all the relevant attributes. Each item has a miss parameter and a guessing parameter, and to ensure the recognizability of the model, it is generally assumed that , i.e., is guaranteed [1,2]. 3. Research Process The core issue of this study is how to construct and apply a Cognitive Diagnostic Models to effectively assess students' practical application skills in Excel data analysis module. First, the core cognitive attributes and skill hierarchy of Excel data analysis module were identified through literature review and expert consultation. Second, a practical test containing Excel data analysis tasks of different difficulties and types was designed to collect data on students' performance in the practical BCP Social Sciences & Humanities ERSS 2024 Volume 23 (2024) 67 exercise. Finally, statistical analysis and data mining techniques were applied to process and analyze the collected data to verify the effectiveness of the Cognitive Diagnostic Models. 3.1 Determine Ability Attributes and Relationships 3.1.1 Determine Ability Attributes Table 1. Core Competencies and Knowledge Structure for Excel Data Analysis Core Competencies and Skills (Attributes) Sub-skills (Test Points) A1 Basic Excel operation skills A1-01 Basic operations of a cell. A1-02 Basic operations of a worksheet (moving, copying, renaming, etc.). A1-03 Workbook saving and renaming. A1-04 Quickly and accurately input data into Excel spreadsheets. A2 Format Setting and Beautification A2_01 Ability to format effectively to better present data and improve readability. A2_02 Setting the data in the cells to a uniform format. A2_03 Ability to use conditional formatting to highlight specific patterns or trends in the data. A3 Data calculation and processing capabilities A3_01 Ability to use formulas and functions to complete the calculation and processing of data. A3_02 Correctly use cell address references to improve data processing efficiency. A3_03 Ability to organize data using sorting and filtering functions to better enable data analysis. A3_04 Ability to recognize anomalous data and process it. A4 Chart creation and analysis A4_01 Ability to select appropriate chart types for data visualization based on data characteristics. A4_02 Consciously adjust chart styles and formatting to improve the clarity of information conveyed. A4_03Ability to select appropriate visualization strategies to present information and trends based on the characteristics of the data and the target audience. A5 Advanced analytical capabilities for data A5_01 Ability to identify key information in data. A5_02 Be able to classify data according to its structure and content, summarize and comparatively analyze different data sets or categories, and interpret the practical significance of these statistics. A5_03 Understand the practical significance and uses of data and apply analytical thinking to data to identify meaningful statistics and meaningless combinations of data. The first step in modeling is to identify the attributes of the competencies and the hierarchical relationships between the competencies. Competency refers to the skills or knowledge that a test taker possesses to successfully accomplish an event. Through in-depth interviews with teachers and industry experts, this study has systematically sorted out the core skills and cognitive attributes that should be mastered by students in the module of “Excel Data Analysis” by closely integrating the teaching objectives, teaching content and students' actual learning situation [3]. After careful analysis and discussion, we identified the following capabilities for the course students need to master the core competencies and skills of Excel: data comprehension A1, basic operations A2, cell formatting CF, data entry and organization A3, data processing A4, data management and analysis A5 and data visualization A6. each core competency contains a number of sub-skills. These sub-skills constitute the cognitive process of students in Excel data analysis, which BCP Social Sciences & Humanities ERSS 2024 Volume 23 (2024) 68 not only covers the basic operational skills of Excel data analysis, but also reflects the understanding of data and advanced application ability, constituting the main dimension of evaluating students' Excel data analysis ability. (see Table 1) 3.1.2 Development of Evaluation Indicators With an in-depth understanding of students' cognitive dimensions and attributes, we develop precise diagnostic indicators and evaluation criteria for each dimension by designing different measurement items. These indicators and evaluation criteria aim to comprehensively reflect students' performance on each cognitive dimension and provide a clear basis for evaluation to achieve a more objective and precise assessment. For example, In the dimension of “basic Excel operation (A1)”, we designed measurement items including filling in information, standardizing the filling of serial numbers, etc., as well as tasks such as copying data and renaming worksheets, etc., to assess students' proficiency in the basic operation skills of Excel, and to examine the standardization and rigor of the operation process. rigor, reflecting students' awareness and ability to improve operational efficiency and their ability to solve problems when faced with them. Through these assessment items, we were able to gain a comprehensive and in-depth understanding of students' overall performance and competence level in basic Excel operations. In the dimension of “Formatting and Embellishment (A2)”, students are assessed on their ability to format data through measurements such as numeric formatting and by examining students' manipulation of fonts, borders, column widths, and other formatting settings, we can assess students' demonstrated ability to enhance the readability and professionalism of data tables. In the “Data Calculation and Processing (A3)” section, we cover Excel formulas and basic functions (such as SUM, AVERAGE, etc.), the use of IF functions and their nesting, the application of VLOOKUP functions, conditional calculation functions, and data sorting and filtering. Among these topics, we also assess the correct referencing of cell addresses, which is crucial for completing data calculations quickly and accurately. The application of formulas and functions tests students' understanding of data relationships, helping them to grasp the quantitative characteristics and business implications of data, laying the foundation for statistical analysis and data interpretation. Data calculation and processing skills are key to ensuring data quality and effective data analysis. Through these specific measurement items, we are able to comprehensively assess the core competencies and basic skills that students should possess after learning the content related to “Excel Data Management and Analysis”. These indicators also provide students with clear learning goals and directions for self-improvement. 3.2 Construction of the DINA Model In this study, the DINA model was used to construct the assessment and evaluation model and to use the model to determine the response patterns of the subjects, so as to provide personalized assessment and guidance. The DINA model is a typical discrete Cognitive Diagnostic Models, which is suitable for cognitive diagnosis of dichotomous item tests, and it describes students as a multidimensional knowledge point mastery vector, and diagnoses students' actual response results. The model emphasizes students' mastery of each knowledge point, and by analyzing students' responses to different questions, it can reveal students' knowledge deficits and provide educators with more targeted teaching suggestions. 3.2.1 Determine the Hierarchy of Ability Attributes After identifying the core competencies of “Excel Data Analysis”, we tried to analyze the relationship between the attributes and clarify the hierarchical relationship between them. In order to more accurately depict the hierarchical relationships between these attributes, we adopted the method of Adjacency Matrix A and Reachability Matrix R (see Table 2). Through Boolean operations, we obtained the optimized Reachability Matrix R, which not only reflects the direct relationship between BCP Social Sciences & Humanities ERSS 2024 Volume 23 (2024) 69 attributes, but also reveals the indirect relationship through transitivity. In the table, “1” indicates that there is a direct or indirect logical relationship between two attributes, and “0” indicates that there is no direct or indirect logical relationship between two attributes. Table 2. Reachability Matrix R Attributes Attributes A1 A2 A3 A4 A5 A1 1 1 1 1 1 A2 0 1 1 1 1 A3 0 0 1 0 1 A4 0 0 0 1 1 A5 0 0 0 0 1 3.2.2 Define the Q-matrix Table 3. Relationship matrix between capability attributes and measurement items Project Attributes A1 A2 A3 A4 A5 1 1 0 0 0 0 2 1 0 0 0 0 3 1 0 0 0 0 4 0 1 0 0 0 5 0 1 0 0 0 6 0 0 1 0 0 7 0 0 1 0 0 8 0 0 1 0 0 9 0 0 1 0 0 10 0 0 1 0 0 11 0 0 1 0 0 12 0 0 1 0 0 13 1 0 1 0 1 14 0 1 0 0 0 15 1 0 0 0 0 16 1 0 0 0 0 17 0 0 1 0 1 18 0 0 1 0 1 19 0 0 1 0 1 20 0 0 1 1 1 21 1 0 0 1 0 22 1 0 0 1 0 The execution of Cognitive Diagnostic Models mostly requires the creation of a matrix with (number of items) rows and (number of attributes) columns. In the matrix, each row represents a test item and each column represents a cognitive attribute. The element in the BCP Social Sciences & Humanities ERSS 2024 Volume 23 (2024) 70 matrix indicates whether the student was able to measure the cognitive attribute upon completion of test item . If , it means that item measured attribute . If , it means that item did not measure attribute . The matrix is a function of the number of items and the number of columns in the matrix. Based on the identified hierarchical relationships of cognitive attributes and skills, in the “Excel data analysis” evaluation, we developed 22 test items, according to which we identified the matrix of relationships between competence attributes and measurement items. (see Table 3). 3.2.3 Ideal Mastery Pattern Ideal Mastery Pattern is the state that a subject should exhibit in the absence of errors and guesswork, and with complete mastery of all relevant cognitive attributes. It provides a target state for assessing a subject's cognitive level. In cognitive diagnosis, a subject's level of mastery of each cognitive attribute can be determined by analyzing the gap between the subject's actual response in answering and the ideal mastery pattern. If a subject's response on a particular test is very close to the ideal mastery pattern, then it can be assumed that the subject has a high level of mastery of the cognitive attribute in question; conversely, if the gap is large, it indicates that the subject is deficient in some of the cognitive attributes. Determining the ideal mastery pattern can be used to assess the subject's mastery of cognitive attributes, which in turn provides goals and directions for instruction and helps teachers target instructional interventions. Upon determining the ideal attribute mastery pattern, combined with the matrix to establish the ideal response pattern, which describes the ideal response behavior of the subjects to the test items. The ideal response pattern is a vector with a length equal to the number of measurement items. For each measurement item, if the student has completed mastery in the corresponding cognitive dimension, the corresponding element in the ideal response pattern is 1, otherwise it is 0. (see Table 4) Table 4. Ideal Mastery Pattern Number Mastering Attributes Ideal Mastery Pattern Ideal Answer Pattern 1 None (0,0,0,0,0) (0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0) 2 A1 (1,0,0,0,0) (1,1,1,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,1,0) 3 A1,A2 (1,1,0,0,0) (1,1,1,1,1,0,0,0,0,0,0,0,0,1,1,1,0,0,0,0,0,0) 4 A1,A2,A3 (1,1,1,0,0) (1,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,0,0,0,0,0) 5 A1,A2,A4 (1,1,0,1,0) (1,1,1,1,1,0,0,0,0,0,0,0,0,1,1,1,0,0,0,0,1,1) 6 A1,A2,A3,A4 (1,1,1,1,0) (1,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,0,0,0,1,1) 7 A1,A2,A3,A4,A5 (1,1,1,1,1) (1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1) 3.3 DINA Model Results Analysis In our study, we utilized the cognitive diagnostic analysis platform (flexCDMs) to implement parameter estimation for the DINA model using the maximum likelihood estimation (MLE) method, thereby evaluating item parameters and students' mastery of various attributes. 3.3.1 Test Evaluation Results Table 5 shows the results of the test parameters calculated. From the analysis results, it is evident that the error rates for questions labeled as Item 4 (border texture settings), Item 5 (date format settings), and Item 17 (data filtering) are relatively high. Through in-depth communication with the students and reviewing their project assignments, we found that many students did not pay sufficient attention to the specific requirements of the questions. Particularly for Item 17, which aims to assess students' mastery of data filtering operations, students only completed single-condition filtering and overlooked the requirement clearly stated in the question to “complete filtering with at least two set conditions.” Therefore, even though they had BCP Social Sciences & Humanities ERSS 2024 Volume 23 (2024) 71 some understanding of filtering operations, these answers were still marked as incorrect. we will strengthen practice on this knowledge point. Table 5. Test Parameters P(0) 1-P(1) Item 1 0.6831 0.2006 Item 2 0.3851 0.2841 Item 3 0.4756 0.3482 Item 4 0.8028 0.1634 Item 5 0.7859 0.3147 Item 6 0.6304 0.1494 Item 7 0.8696 0.0779 Item 8 0.6957 0.1688 Item 9 0.239 0.0195 Item 10 0.1086 0.1234 Item 11 1.00E-04 0.0065 Item 12 1.00E-04 0.013 Item 13 0.3826 0.3366 Item 14 0.0017 0.0804 Item 15 0.468 0.0676 Item 16 0.6359 1.00E-04 Item 17 0.7236 0.0441 Item 18 0.508 0.0751 Item 19 0.3117 0.3371 Item 20 0.3355 0.0244 Item 21 0.0358 0.1868 Item 22 0.0741 0.221 On the other hand, questions like Item 11 and Item 12 (applications of basic functions) and Item 20 (creation of charts) have a lower error rate. This indicates that once students master the relevant knowledge, they are generally able to answer these questions correctly, and the accuracy of these questions is relatively high. The results of the comprehensive evaluation indicate that items 11, 12, 9, and 20 exhibit lower parameters for guessing and errors. These items excel in distinguishing the extent of knowledge mastery among examinees and are capable of accurately assessing the true ability levels of the examinees, thus indicating high item quality. Only when candidates truly master the relevant knowledge can they answer questions correctly; conversely, if candidates have not mastered the knowledge, it is difficult for them to guess correctly. However, items such as Item 4, Item 5, Item 7 have higher error parameters, while items like Item 2, Item 3, Item 5 have higher guessing parameters. These items require further optimization in the future, considering improvement measures from multiple perspectives to enhance project quality. 3.3.2 Pattern Discrimination In the study, by applying the flexCDMs platform, we successfully evaluated the probability of students mastering each attribute. Overall, there were significant differences in the mastery levels of students across five attributes (A1, A2, A3, A4, A5). In particular, the mastery of attribute A3 (data computation and processing) was particularly prominent, with many students achieving a probability of 1 or close to 1 for this attribute, indicating that compared to other attributes, students have a more solid grasp of knowledge or skills in this area. Attribute A5 corresponds to students' understanding BCP Social Sciences & Humanities ERSS 2024 Volume 23 (2024) 72 of data and advanced analytical skills, which is not a simple knowledge point but requires a certain amount of training for students to acquire. The analysis results revealed that students' mastery levels for attribute A5 were generally not high, with many students' mastery levels for this attribute only reaching medium or below compared to other attributes. This finding has important guiding significance for adjusting teaching strategies and methods, suggesting that we should focus on those attributes with insufficient mastery and take measures to strengthen students' cultivation in data thinking and analytical abilities. We set a clear threshold for the mastery probability of each attribute based on the attributes of students and the difficulty level of the attributes. The threshold for attributes A1 and A2 is set at 0.9, while for attributes A3, A4, and A5, the threshold for mastery is set at 0.85. If the mastery probability of an attribute is higher than the threshold, it is considered mastered and marked as “1”; if it is lower than the threshold, it is considered not mastered and marked as “0”. We have conducted detailed statistics on the mastery patterns of students. (see Table 6) Table 6. Summary of patterns Pattern Number of students Percentage 0,0,0,0,0 12 6.00% 0,0,1,0,0 7 3.50% 0,1,1,0,0 21 10.50% 1,0,0,1,1 1 0.50% 1,0,1,0,0 16 8.00% 1,0,1,1,0 8 4.00% 1,0,1,1,1 4 2.00% 1,1,0,0,1 13 6.50% 1,1,0,1,1 20 10.00% 1,1,1,0,0 6 3.00% 1,1,1,0,1 10 5.00% 1,1,1,1,0 8 4.00% 1,1,1,1,1 74 37.00% 4. Summary This study successfully evaluated the effectiveness of item development and students' mastery of multiple key competency attributes in the “Excel Data Analysis” module by constructing and applying the DINA model. The results indicated that students performed well in basic operations and data computation processing, but there is still room for improvement in their advanced data analysis capabilities. Our research not only provided targeted teaching feedback for teachers but also offered data support for adjusting teaching strategies and designing personalized learning paths for students. Despite the limitations in sample size and the scope of model application, this study laid the groundwork for applying cognitive diagnostic models in a broader educational context in the future. Future research should consider expanding the sample size and further validating the stability and universality of the model to achieve more comprehensive assessment and optimization. Acknowledgments 2022 Annual Project of Shaanxi Provincial Education Science “14th Five Year Plan” (SGH22Y 1803). BCP Social Sciences & Humanities ERSS 2024 Volume 23 (2024) 73 References [1] Tu Dongbo, Cai Yan, Gao Xuliang, et al. Advanced Cognitive Diagnosis [M]. Beijing: Beijing Normal University Press, 2019, p.22-23. [2] Xu T, Wu X, Sun S, et al. Cognitive diagnostic analysis of students' mathematical competency based on the DINA model[J]. Psychology in the Schools,2023,60(9),p.3135-3150. [3] Wang Fengsi, et al. Educational Big Data -- Assessment Evaluation Data Analysis, Mining, and Application [M]. Beijing: Science Press, 2020, p. 61-66. [4] Hang Dandan. Research on the Validation Method and Application of Q Matrix in Multidimensional Cognitive Diagnostic Assessment [D]. Jiangxi Normal University, 2020. DOI: 10.27178/d. cnki. gjxsu. 2020.001433. [5] Dai Haiqi, Zhang Qinghua. Application Study of Rule Space Model in Descriptive Statistical Learning Pattern Recognition [J]. Psychological Science, 2004, (04), p.949-951+948. DOI: 10.16719/j.cnki.1671- 6981.2004.04.048. [6] Jianhua X, Zhaosheng L, Guanzhong L, et al. Data-driven Q-matrix learning based on Boolean matrix factorization in cognitive diagnostic assessment. [J].The British journal of mathematical and statistical psychology, 2022,75(3),p.638-667.