EXPERT SYSTEMS VERSUS TRADITIONAL METHODS FOR TEACHING ACCOUNTING ISSUES Developments In Business Simulation & Experiential Exercises, Volume 19, 1992 131 EXPERT SYSTEMS VERSUS TRADITIONAL METHODS FOR TEACHING ACCOUNTING ISSUES Marcus D. Odom, University of Southwestern Louisiana David S. Murphy, Oklahoma State University ABSTRACT The use of expert systems as pedagogical devices has received little attention in the accounting literature. This study reports on an experiment that was designed to assess the effects of expert system use on student learning. The cognitive theories of learning were drawn on to develop a framework for measuring learning when expert systems are used as pedagogical aids. Two components of leaning, declarative knowledge and procedural knowledge were measured. Treatment groups used either an expert system, received instruction, or used an expert system and received instruction. The results indicate that instruction facilitated the development of declarative knowledge to a greater extent than the use of an expert system. Further, the results indicate that expert system use facilitated the development of procedural knowledge to a greater extent than the use of instruction. Thus, although students who learned transfer pricing using an expert system outperformed students who had learned the same subject matter through instruction in a procedural test, the expert system users did not appear to understand why the procedures they followed were effective. INTRODUCTION The use of expert systems as pedagogical devices has received little attention in the accounting literature. Expert systems are sophisticated computer programs, which are designed to replicate an expert’s judgment in a modeled domain. Consequently, nonexperts who use expert systems have the opportunity to watch an “expert” at work. Böer and Livnat (1990) in a study of the effect of expert system use on homework assignments concluded that expert systems seemed to have the potential to enhance the learning process for accounting students. However, their experiment may not have measured student learning but may have instead observed the phenomenon that nonexperts who use expert systems make better decisions than nonexperts who do not. The study herein reported draws on cognitive theories of learning to develop a framework for measuring learning when expert systems are used as pedagogical aids. A cognitive model of learning is presented in the next section and is then used to derive the research hypotheses. The research design and empirical results are then presented. The results are discussed in the conclusion, as are implications for accounting and business education and suggestions for future research. COGNITIVE MODEL OF LEARNING The cognitive model of learning is based on the processes, which affect data as the data flows through the human information processing system. When data enters the processing system, it is transformed into knowledge, which is stored in long-term memory. The type and amount of knowledge stored in long-term memory increases during learning. Two types of knowledge are stored in long-term memory: declarative knowledge and procedural knowledge. Declarative knowledge is factual knowledge about a topic (Gagné, 1985). Declarative knowledge is instrumental in developing procedural knowledge, the knowledge of how to do something. Anderson (1985, p. 198) describes procedural knowledge as “ . . . knowledge about how to perform various cognitive activities.” Both declarative knowledge and procedural knowledge are developed as an individual learns a new skill. Fitts and Posner (1967) identify three stages in skill development: the cognitive, associative and autonomous stages. Declarative knowledge, knowledge about facts and things, is developed in the cognitive stage, as are domain- general problem-solving strategies. Domain-specific problem solving procedures are developed during the associative phase as the individual corrects and refines the domain-general strategies. This refinement process occurs as new information interacts with developed knowledge. No clear distinction between the associative and autonomous stages exist. The speed and accuracy with which procedures are used increases as individuals move into the autonomous stage. Individuals generally move sequentially through all three phase as they develop a new skill. Learning is measured in this study by measuring the development of declarative and procedural knowledge of the subjects. The amount of knowledge that the students have at the beginning of the experiment is important in measuring the earning that can be attributed to the treatments. According to the cognitive model, declarative knowledge and procedural knowledge are developed when prior knowledge interacts with new information. This study compares the affects of instruction and expert system use on student learning. Learning results in the development of both declarative and procedural knowledge. However, different instructional technologies may have different effects on the type and extent of knowledge developed. The presentation of factual information should result in the development of declarative knowledge because declarative knowledge is knowledge about facts and things. The presentation of factual information should, on the other Developments In Business Simulation & Experiential Exercises, Volume 19, 1992 132 hand, have a limited effect on the development of procedural knowledge. Factual information by definition is information about things, not information about how to do things. Expert systems guide users through a decision-making process. This process demonstrates a procedure for solving a problem but may not simultaneously provide factual information about entities. Students, as they develop expertise in a target domain should be able to use their procedural knowledge to solve relevant problems, and also be able to explain or describe underlying concepts and facts. Thus, a viable instructional technology should facilitate the development of both declarative and procedural knowledge and will most likely guide students through the three stages of skill development (Fitts and Posner, 1967). This experiment compares the relative effectiveness of instruction and expert system use on the development of declarative and procedural knowledge. The hypotheses to be tested are: H1: There will be no difference in development of declarative knowledge between treatment groups. H2: There will be no difference in development of procedural knowledge between treatment groups. EXPERIMENTAL METHOD Task Subjects completed a set of transfer pricing cases in this experiment. Transfer pricing has been suggested as a viable topic for expert system-based instruction (Böer and Livnat, 1990). Transfer pricing decisions required the analysis of a complex set of rules. However, these rules can be readily divided into small segments and then analyzed in sequence during the decision process. The rules for the transfer pricing expert system were derived from Thomas (1989) matrix approach to transfer pricing. The rules were implemented using a rule-based expert system shell using a series of IF-THEN statements. The subjects had the option of viewing the rules and the reasons for the queries made by the expert system during a consultation session. The subjects could also look at a complete explanation of the solution path upon completion of the session. Subjects Ninety-seven students from three sections of undergraduate managerial accounting taught by the same instructor participated in this study. This course is a required course for several majors at the sophomore level. Treatments were randomly assigned to course sections, and not individual students. Consequently, group sizes were uneven. Demographic Questionnaire To test for homogeneity of the sections, the students completed a demographic questionnaire. The variables on the questionnaire were tested for use as covariates in the analysis. Pretest All three sections were given a pretest to measure their knowledge of transfer pricing prior to the treatment. The pretest consisted of ten questions designed to measure the student’s declarative knowledge. Base-level procedural knowledge was measured by having the students solve six transfer-pricing problems. The subjects were required to act upon the information presented that is to use procedural knowledge, in solving these cases. Both pretest elements were conducted without the use of any learning aids. Cases The subjects were then given a set of transfer pricing cases to solve after they had received instruction and/or an expert system. The assigned cases became progressively more difficult to maximize the benefits of the learning experience. The subjects completed a posttest after finishing all of the experimental cases. The posttest was given one week after the pretest and two days after completion of the experimental cases. Posttest The posttest, given to all three sections, consisted of the same ten questions that were used in the pretest to measure declarative knowledge. However, the order of the questions on the posttest differed from the order on the pretest. Seven transfer pricing problems were given to the students to measure their new levels of procedural knowledge. The posttest was completed by all three sections without the use of any type of decision aid to ensure that learning, and not tool use, was being measured. Independent Variables Two independent variables were manipulated in this study: instruction and expert system use. Instruction was manipulated by presenting two sections with the matrix approach to transfer pricing. This presentation consisted of lectures and problem solving demonstrations using the matrix approach. Expert system use was manipulated by providing two sections with a transfer-pricing expert system. The use and operation of the expert system was demonstrated in class and the subjects observed the instructor solving sample problems using the expert system during class. One section received instruction only (21 subjects), another section received an expert system only (41 subjects). The third section (35 subjects) was provided with both the transfer pricing expert system and a discussion of the matrix approach. Thus, the third section was presented with two approaches to solving transfer-pricing problems. Several copies of the expert system were Developments In Business Simulation & Experiential Exercises, Volume 19, 1992 133 available to the subjects in the computer lab for their use in solving the experimental cases. Dependent Variables Two variables were used to measure the development of the subjects knowledge over the course of the experiment. The development of declarative knowledge was measured as the percentage of correct answers on the multiple-choice posttest. The percentage of correct answers on the posttest transfer pricing problems was used as a measure of procedural knowledge. Pretest scores were used as covariates in the ANCOVA models. Cook and Campbell (1979) note that gain score analysis, the use of the difference between pretest and posttest scores as the dependent variable, is generally less precise than covariance analysis. Gain score analysis does not test for interaction effects between the pretest and treatments. RESULTS Subjects Subject demographics are presented in Table 1. As discussed above, treatment group sizes were unequal. However, the demographic factors appear to be similar across treatment groups. A Tukey HSD test indicated that mean GPAs between groups were not significantly different (alpha = 0.05, critical range = 0.332). Nevertheless, gender and class (sophomore, junior, senior) were used as covariates in the following analyses. Hypothesis 1 The first hypothesis postulated that the experimental treatment would not have a significant effect on the development of students’ declarative knowledge. Table 2 shows the differences in pretest and posttest multiple choice scores by subject. The differences between mean pretest and posttest scores for the expert system with instruction and the instruction only treatment groups appear greater than that of the expert system treatment group. Moreover, subjects generally exhibited an increase in perceptual cohesiveness (reduced standard deviation) as a result of treatment application. The results of the ANCOVA test of hypothesis 1 are presented in Table 3. This test indicates that instruction had a significant (F=3.342, df=1, 87, p=.O71) effect on the development of declarative knowledge as measured by the multiple choice instrument and that pretest scores had a highly significant (F=1.716, df=1, 87, p=.000) effect. Neither expert system nor the other covariates (gender and class) had significant effects on posttest scores. Consequently Hi is rejected. Developments In Business Simulation & Experiential Exercises, Volume 19, 1992 134 Hypothesis 2 The second hypothesis postulated that the treatments would not affect the development of procedural knowledge. The development of procedural knowledge was measured by the percentage of correct responses to a series of transfer-pricing cases. Mean percentages by treatment are presented in Table 4. Subjects who used the expert system showed slightly higher posttest scores and greater differences between pretest and posttest scores than the other treatment groups. Standard deviations for all groups decreased after the treatment indicating an increase in response consensus. The differences between pretest and posttest scores within all of the groups appears to be significant. Treatment differences were tested in an ANCOVA model. Posttest score was the dependent variable and pretest score, treatments, gender and class were treated as independent variables in the analysis. The results of the ANCOVA test are presented in Table 5. As shown in Table 5, expert system use had a significant effect (F=7.722, df=i, 87, p=0.007) on the development of students’ procedural knowledge, as did class and the pretest scores. Instruction did not have a significant effect on posttest scores (F=122.552, df=1, 87, p=0.270). Consequently, H2 is rejected. Subjects who had used the expert system showed both the highest mean posttest score and the greatest gain score. Students who received instruction only showed the lowest mean posttest score and the smallest gain score. CONCLUSION This experiment was designed to assess the effects of expert system use on student learning. Two components of learning, declarative knowledge and procedural knowledge, were measured. Treatment groups used either an expert system, received instruction, or used an expert system and received instruction. The results indicate that expert system use facilitated the development of procedural knowledge. Further, the results indicate that expert system use did not facilitate the development of declarative knowledge to the same extent, as did instruction. Thus, although students who learned transfer pricing using an expert system outperformed students who had learned the same subject matter through instruction in a procedural test, the expert system users did not appear to understand why the procedures they followed were effective. It appears that expert system use may have “jump started the skill development process because the expert system users did not appear to develop the same level of declarative knowledge. Instruction appears to facilitate the development of declarative knowledge and to Developments In Business Simulation & Experiential Exercises, Volume 19, 1992 135 provide answers to “why?’ questions while expert system use, or learning by example, provides answers to ‘how?” questions. These results are consistent, for the most part, with Murphy (i990) who demonstrated that expert system use had an adverse effect on the development of subjects’ semantic memory (declarative knowledge). Consequently, it appears that expert systems may function as viable instructional aids, but that they should not be used as replacements for factual instruction. Research is needed to determine if these findings are generalizable to a larger class of learning problems. In addition, research is needed to assess the long-term effects of expert system use on the development of both declarative and procedural knowledge. Research should be conducted to assess the effect of expert system explanations on the development of declarative knowledge. This research suffers from the same limitations as most laboratory experiments but should provide guidance for others considering the use of expert systems as pedagogical aids. The use of three courses taught by the same instructor controlled for instructor effects but limits the generalizability of the findings. An instructor can provide the detailed or declarative knowledge about the target domain while the expert system will provide the students with an “expert and the opportunity to learn by example. REFERENCES Anderson, J. R. (i985) Cognitive Psychology and it Implications, 2nd Ed. New York: W. H. Freeman and Company. Böer, G. B. and J. Livnat (1990).“Using Expert Systems to Teach Complex Accounting Issues”, Issues in Accounting Education, 5, 108-i i9. Cook, T. D. and D. T. Campbell. (1979). Quasi- Experimentation: Design & Analysis Issues for Field Settings. Boston: Houghton Mifflin Company. Fitts, P. M. and M. I. Posner (1967) Human Performance Belmont, CA: Brooks Cole. Gagné, E. D. (1985. The Cognitive Psychology of School Learning Boston Little Brown and Company. Murphy, D.S (1990) “Expert System Use and the Development of Expertise in Auditing: A Preliminary Investigation, The Journal of Information Systems, 4, 18-35. Thomas, M. F. (i989) ‘A Matrix Approach to Transfer Pricing’, Oklahoma State University, School of Accounting, working paper. Table of Contents Volume 19, 1992 The Pedagogical Utility of a Management Simulation Game in a Business Policy Course Modeling Economic Environments in Business Simulations: Some Comparisons and Recommendations Experiential Learning and TQM Principles: Teaching Behavioral Science in a Business School The Use of Poster Presentations as a Final Project in the Business Policy Course The Device Business: A Management Simulation for Strategy Formulation The Advantages of Experiential learning in the Auditing curriculum A Framework for the Identification of Moderated and Mediated Performance consequences of Pedagogical Alternatives Simulating Qualitative Research Relating to Values and Lifestyle Segmentation A New Market Demand Model for Business Simulators Interactive Optimization Using the Method of Relative Improvement Preferences: Methodology and Empirical Evaluation The Service Trainer Simulation Benefits of Internet Computer Networks for ABSEL Members How Should We Measure Experiential Learning? An Assessment of Simulation Usage in Management Accounting Courses The Influence of Myers-Briggs Type and Group Dynamics on Simulation Performance Effective Leadership Behavior in the Desert Storm Arena: An Application of the Vroom-Yetton-Jago Model Installing and Consolidating Work-Team Values: The Effects of a Multicultural Outdoors Experiential Program Key Determinants and Decision performance in a Business Simulations and Experiential Learning environment Multi-Cultural Experiential Learning: A Computer Simulation in Indonesia Attitudes Toward and Emotions Related to Women as Managers: A Replication and Beyond Scenario Approach to Simulating Consumer Expenditures: A Cross-Cultural Analysis Insights into Ethical Decision making Activities and Organizational Performance: A Management Simulation Analysis of College Students and Managers Evaluating a Business Simulation Program for Joint Venture Negotiation and Management Teaching Business Interviewing Strategies with an Experiential Approach Cooperative Learning Across the Business Curriculum Modeling Total Quality Elements into a Strategy-Oriented Simulation Teaching Business Decision making using a Simulator Extending the Educational Utility of a Simulated Competition within the Confines of an Established Undergraduate Marketing Curriculum Expert Systems Versus Traditional Methods for Teaching Accounting Issues Confidence Extremes Diminish Quality Performance in a Total Enterprise Simulation Can Ethics Be Taught? A Simulation Tests a Traditional Ethics Pedagogy Through The Looking Glass, Inc.: Organizational Climate Research as Experiential Pedagogy The use of Cluster Analysis for Business Game Performance Analysis Power and Ethnicity: An Experiential Learning Exercise (How to Sensitize Students to Diversity) Directed Development of Critical and Creative Thinking Skills for Case Analysis Implementing Total Quality Management in a Computerized Business Simulation Product Quality in Business Simulations Use of Simulation for Ethics Education in Management Satisfying the University's Customers through Total Quality Management Instruction: A Case Study Personality Characteristics and Group Performance in Total Enterprise Simulations Concepts of Interval Estimation and Quality Control Charts via Computer Simulated Sampling An Examination of the Effect of Team Cohesion, Player Attitude, and Performance Expectations on Simulation Performance Results The Effectiveness of Inventory Management and Production Scheduling Training in a Total Quality Management Environment Peer Group Indicators of the External Validity of Business Games: A five-year Longitudinal Study Political Strategies and Personal Actions Does Practice make Perfect? Observations on Simulation Trial The Use of A Non-Business Computer Simulation to Teach Marketing Management BankPro Commercial Bank Simulation The Production Game Pursuing Excellence: Work Strengths, Assets and relevant Values (An Exercise) Test of a Short Outward Bound Experience for College Students What is it that we want Student To Learn? Using Two TQM Philosophies when Playing Blackjack The Lagged Effects of Decision Variables on Financial Performance Measures Used in Two business Simulations The Development of an Experiential Exercise for Career Planning and Effective Job search Performance Measuring Quality in Management of Business Pedagogy Exploring Quality and Productivity Improvement: Using and Experiential Process Toward a Generalized Architecture for Intelligent Reactive Management Systems The Quality Game TQX: Using Expert Systems to Improve Training in Total Quality Management A Simulation of the Effect of the Medicaid Payment Lag on the Financial Position of Community Pharmacies The Use of a Board of Directors to Evaluate and Validate Decisions in a Competitive Graduate Management Simulation Course The Impact of Academic Dishonesty on Business Simulations and Experiential Learning Activities Picture Project: An Experience of Icebreaking and/or Decision-Making An Effective Role-playing Exercise for Teaching Requisite TQM Supervisory Attitudes/Behavior Computerized Management Simulations and Some Correlates of Students' Satisfaction The Timing and Stability of Reactions to Market Structure in a Single Player Simulation Environment A Graphics Application Extension for a Simulated Decision Support System Environment Aspects of a Group Project Utilizing Actual Business data and a Computerized Accounting System The Role of Universities' Extended Learning Department in Assisting Organizations Implement Total Quality Management A three-dimensional Learning Experience to Develop Total Quality Management Skills Estimating Quality Costs by Computer Simulation Applications and Examples of Quality Control Software Assessing Business Pedagogy A Demonstration of an Experiential Process for Exploring Quality and Productivity Improvement Processes An Action-Ethics Dilemma: A Demonstration Org Sim Jr.: A 2-3 Hour Version of the 2-Day Blanchard/Murrell Organization Simulation Building Buildings: An Experiential Exercise in Organizational Structure, Communication, Leadership and Group Dynamics Expert Systems for Organization Design: A Demonstration A Demonstration of Product Quality in a Business Simulation: Version 2 of CEO Two Revolutions, Total Quality Leadership, and the Baldy: The Story of Milliken Total Quality after the Award - The Xerox Story Building a Competitive Advantage through Customer Satisfaction and Reengineering Human Resource Planning: Managing the Only Renewable Resource for a Competitive Advantage Using Simulations to Teach International Issues: An analysis of the Multinational Management Game's Learning Environment Quality Function Deployment: A Tutorial Care and Nurturing of Teams Making a Good Thing Better: Adding TQM to Participative management Empowering Organizations to Redesign and Transform Themselves - Eastman Chemical Using Simulations in Field Management Development of an International Life Insurance Company Reinforcing the TQ Environment via Simulation Transforming a Business College into a Total Quality College