USING AN EXPERT SYSTEMS BASED DECISION AID IN ACCOUNTING INFORMATION SYSTEMS Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 35 USING AN EXPERT SYSTEMS BASED DECISION AID IN ACCOUNTING INFORMATION SYSTEMS Patrick B Dorr, Oklahoma State University Martha M. Eining, University of Utah James E. Groff, Oklahoma State University ABSTRACT Accountants have shown considerable interest in adapting expert systems for use in accounting practice. Accounting courses have not vet integrated this material into the accounting curriculum. In this paper, we suggest a method which can be used to develop an accounting expert system decision aid for classroom use. We compare the effects of using an expert systems based decision aid with a more traditional decision aid (a check list) accounting information systems classes. Our evidence indicates that the expert systems decision aid was a more effective learning aid than was the checklist. The students found it to be easy to use and helpful in learning the internal control concepts being presented. INTRODUCTION Professional accountants are making a significant effort to adapt expert systems technology to accounting practice. Accounting literature contains descriptions of a number of systems that have already been developed for tax and auditing applications (Dungan & Chandler [1985]. Hansen & Messier 11986), Michaelsen [1984] and Steinbart [1987]). A number of public accounting firms have made major commitments to the development of expert system decision aids. An example of one of these is Coopers & Lvbrand’s ExperTAX (Newquist [1987]). The interest on the part of practitioners in expert systems applications, however, has not been mirrored in the classroom. This is in part because few systems are generally available and the commercially available systems are complex and costly. Also because most academic accountants lack expert systems expertise, they hesitate to attempt to develop their own systems for classroom use. In this paper we suggest a method for developing an expert system which can be used by an instructor without significant commitment to developing artificial intelligence or expert systems tools. The expert system (ES) described in this paper deals with the evaluation of internal controls over payroll processing. The ES was utilized in undergraduate accounting information systems classes as a means of exposing the students the latest technological innovations in decision support aids. The exercise was also designed to serve as supplementary training for enhancing the student’s skill in one of the courses primary topics, the evaluation of internal control. The performance of the students using the ES decision aid was compared to that of students using a more traditional decision aid a check list 1 and to that of students using no decision aid. In Post-Practice evaluations, the performance of the ES group was superior to either of the other two groups. Although the ES presented, like several others reflected in the academic literature, deals specifically with the evaluation of internal control, the approach is general enough to be applied to a number of accounting problems. Normally a good candidate for expert system modeling is a problem which is not trivial and cannot be solved with common sense. Such a problem would likely require an expert several minutes to a few hours to arrive at a decision. Also, as suggested by the many ES applications in auditing, a ES problem commonly requires a mix of objective and subjective inputs in reaching a decision. Before discussing the ES developed for class purposes, the reader is provided with a brief background on expert system concepts. In addition, information is presented on relatively inexpensive software which can greatly aid in the efficient development of an ES for classroom use.1 EXPERT SYSTEMS BACKGROUND ES technology has grown out of research in artificial intelligence. An expert system is a computer program which attempts to replicate the decision making process of an expert. The two major components of any expert system are the know ledge base and the inference engine. The knowledge base consists of the specialized knowledge an expert uses to formulate a decision. This knowledge base is represented by a set of IF/THEN/ELSE rules which reflect the experts responses to identified cues. The inference engine represents the problem-solving method which guides the application of the rules to arrive at a decision approximating the quality of the expert’s decision. To build an expert system, the designer must develop both the knowledge base and an inference engine which is appropriate for the problem being addressed. Early attempts to develop expert systems involved identifying an expert in the field of the problem, soliciting from the expert information for the knowledge base and then writing a computer program for the inference engine. These programs were usually written in highly specialized languages, such as LISP or PROLOG, which have properties which make them particularly well suited to this type of problem. The highly specialized nature of this procedure made expert systems technologically inaccessible to those lacking substantial training and/or resources. 1 Several ES software packages including the one this research are available commercially for less than $500. Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 36 EXPERT SYSTEM SHELLS Accessibility to ES technology has been significantly enhanced with the development of ES shells. ES shells are computer programs which separate the problem of acquiring the knowledge base from that of developing the inference engine. - Since decision making methods do not vary significantly across large classes of decisions, a common type of inference engine2 can be used with many knowledge bases. The ES shell consists of a program which develops the inference engine for the knowledge base being used and a mechanism for entering the knowledge base obtained from the expert. In addition, most commercially available ES shells have facilities for interfacing with other software such as Basic and Lotus 1-2-3. This significantly increases accessibility since many potential ES developers may be quite familiar with the knowledge base but lack the necessary programming skills required to develop the inference engine. The developer’s problem then becomes one of selecting the ES shell which best suits the decision maker and the problem being modeled. Several types of ES shells are commercially available. The two most common types are: rule-based shells. such as VP- Expert, EXSYS and Personal Consultant Easy; and example- driven shells, such as 1st-Class and Expert Ease. Rule-based shells require the user to input the knowledge base using a series of IF/THEN/ELSE rules. Example-driven ES shells require the developer to identify the significant factors which impact a decision then input levels of those factors and outcomes using actual decisions made by an expert. The knowledge base is built up by the ES shell program from those decision cases. Rule-based shells require somewhat more experience on the part of the ES developer since he/she must become familiar with the IF/THEN/ELSE method of formulating the knowledge base. They also may take more of a commitment on the part of the expert since the expert must be queried to determine the rules. One of the major costs in developing ExperTAX was the cost of expert time (Newquist [1987]). Also, experts may not be able to quantify their knowledge into such a rules-based format (Hanson & Messier [1986]). Rule-based ES shells are particularly useful when applied to problems which have some outcomes which occur infrequently. A randomly chosen set of actual decisions would be expected to include few or no cases with these outcomes. Rules leading to these outcomes may not be developed using an example-driven shell because these outcomes are rare in actual data. However, the expert recognizes that rare outcomes can occur and consequently includes rules which lead to those outcomes. Once the knowledge base has been entered into a rule-based shell as a series of IF/THEN/ELSE rules, the shell uses an algorithm to develop the inference engine. The most common algorithm used to develop the inference engine is backward chaining. In backward chaining the program starts with an outcome or goal variable and works backwards through the rules to develop decision paths to each outcome.3 2For a description of a number of commercially available ES shells see Olsen et al. [1987]. 3 For a discussion of an application of a rule based ES shell to a tax problem see Thompson & Thompson [1987b]. Example-driven ES shells provide an alternative when the knowledge base is not easily formulated into IF/THEN/ELSE rules or the developer does not wish to invest the time necessary to develop the skills required to formulate the knowledge base in that manner. An example- driven ES shell requires the system developer to specify the decision outcomes or results and the attributes or factors used by the expert to arrive at a result. Both factors and results may be qualitative or quantitative. The developer then supplies values for factors and results from actual cases in which an expert has made decisions. The ES shell then inductively develops the decision rules, in the form of a decision tree, from the examples provided by the developer. Example-driven ES shells may not require as much of a time commitment by the expert as rule-based shells. If decision examples already exist they can be used to develop the knowledge base. In that case the expert may be used to isolate the factors and their levels. One major constraint imposed by example-driven ES shells is that the number of decision paths increase geometrically with the number of factors. This can make the decision tree cumbersome and difficult to understand. Since more examples are required as the number of factors and levels for these factors grow, it may be difficult to obtain a sufficiently large number of examples to develop an ES which can provide valid decisions. Also, as suggested above it is difficult to develop decision paths to rare outcomes with example-driven shells.4 DEVELOPING THE ES DECISION AID FOR CLASS USE The objective of this experiment was to develop and test an ES which would be Used in an upper division undergraduate Accounting Information Systems class to expose accounting students to ES technology. Several criteria were established far the ES decision aid. First, the ES had to be sophisticated enough to provide the type of experiences students might have when dealing with ES decision aids in practice. Second, the ES had to deal with a decision problem familiar to the students. Finally, the ES had to be developed with limited resources--both of time and money. The 1st-Class5 example-driven ES shell was chosen as the package which best satisfied the above criteria. The ES decision aid was to be developed for classroom use only and no effort was made to make ii a commercially viable tool. This significantly reduced the time and cost involved since we did not commit 4 For a discussion of example driven ESs see Thompson & Thompson [1987a]. 5 1st-Class is a product of Programs In Motion Inc. , 10 Sycamore Rd., Wayland, MA 01778. It is available for under $500. We also required software which would run under MS-DOS which 1st-Class does. Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 37 resources to validating the ES on a different set of cases from those used to develop it. If the ES was going to be used on a new set of problems then validation would have been necessary. At the present time there appears to be little consensus as to what constitutes an acceptable approach to testing the validity of an ES.6 The evaluation of internal control in a manual accounting system was chosen as the problem to be addressed by the expert system for several reasons. As mentioned earlier, internal control has not only been the focus of auditing research concerning decision-support and expert systems, it is also one of the central topics covered in the undergraduate accounting information systems course. The evaluation of internal control represents a highly complex decision process, filled with subjectivity and requiring multiple judgments of a qualitative nature. Such a decision process has not been effectively modeled using traditional decision support systems and, thus, is ripe for expert system treatment. Also, because of previous course coverage of internal control, all the students should have been familiar with this type of problem and the factors involved in making such an evaluation. The subjective nature of internal control assessments also suggested that the use of the expert system could possibly provide valuable reinforcement of principles which are dealt with at a more conceptual level in the classroom. IDENTIFICATION OF OUTCOMES AND FACTORS The first step in developing an ES using an example-based shell is identification of outcomes and factors. An outcome is the result, conclusion or goal to be provided by the ES. For our ES the outcome was an assessment of the adequacy of internal controls in a manual payroll system. while both qualitative and quantitative values for the outcome are possible, internal control in this ES was assessed on symmetric scale from 0 (total absence of controls) to 100 (all possible controls are present). Factors are the elements involved in arriving at a result. Developing a set of factors is probably the most difficult part of using an example-driven ES shell. While an expert may be able to contribute to the identification of the set of factors, m individual expert may not be able to identify specific factors or may produce an incomplete set. An alternative is to develop the factors using authoritative sources. For our ES a list of twenty-five factors which contribute to the assessment of the quality of internal controls over payroll was developed using the CONTROLPLAN internal control questionnaire developed by Deloitte. Haskins & Sells. This list was reviewed for completeness by consulting current auditing texts and a survey of several auditing faculty. These factors along with their levels are presented in the Appendix I. Example-driven ES shells are most useful when a sample of decisions which have already been made is available. The levels of the factors and the outcomes are entered and then the decision tree is developed inductively from those examples by the inference engine. For the types of narrowly focused problems addressed in a class setting a large sample of decision data is not generally available. As an alternative, to obtain the case data, fifty scenarios were written in which the levels of the factors were manipulated. A sample scenario is shown in Appendix II. No attempt was made to obtain a complete factorial arrangement of the factors because the number of scenarios necessary to do so would have been prohibitively 6 For a discussion of validation of accounting ESs see OLeary [1987]. large. These fifty scenarios were presented to an audit manager of a Big Eight accounting firm with over eight years audit experience who had agreed to act as the expert. He was asked to read each scenario and provide an assessment of the controls on payroll using the 0-100 scale. The factor values in each scenario and the experts assessment were then entered into the 1st-Class ES shell. After the fifty cases were entered the ES shell inductively generated an optimized decision tree, implicitly determining the intermediate nodes. The 1st-Class package then produced a user interface in the term of a series of multiple- choice screens as depicted in Figure 1. The first question asked is always the one at the top of the rule presented in the decision tree developed by the 1st Class inference engine. The order of all later questions is driven by the response to the previous question. Once 1st-Class has obtained all Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 38 the necessary responses to its queries, it provides the user with a suggested evaluation and an explanation of the underlying rule. A sample advisor screen is shown in Figure 2. AN EVALUATION OF THE ES DECISION KID A total of 1917 junior and senior students enrolled in Accounting Information Systems were used to test the ES decision aid. To establish a base level of performance all students were given five scenarios chosen from the fifty evaluated by the expert and asked to evaluate the quality of the internal control using the 0-100 scale. These students did not have access to any decision aid. The five scenarios used were chosen from the scenarios which were assigned scores between 25 and 75 by the expert. This avoided scenarios in which the choice appeared to be obvious to the students. 7 Two hundred and fifteen students began the experiment. Fifteen dropped the course, seven railed to complete one or more sessions and two and equipment problems which invalidated their responses. These missing subjects were randomly distributed among the groups and therefore are not likely to have affected the results. For purposes of the experiment, students were randomly assigned to one of four groups as shown in Table 1. The only difference between the two ES groups was that the ES used by the With Explanation group had the capability of providing an explanation of the reasoning behind the decision if the user requested it. An analysis of responses to a demographic questionnaire indicated no systematic differences among the groups. Each student then participated in three practice sessions o{ approximately one-hour each. In the practice session each student was given eight scenarios to evaluate. Those in the decision aid (ES and questionnaire) groups used their decision aids. Those in the control group read and evaluated the scenarios without the use of a decision aid. These practice sessions occurred over a period of approximately four weeks. During a fifth session, the students were given the same five scenarios used in the pre-practice session to evaluate, however, their order was randomized. Again during this session the students did not have access to a decision aid. The students were also given a post experiment questionnaire. RESULTS As a test of performance the mean absolute difference (MAD) between the experts and students’ assessments were computed for the pre- and post-practice sessions. The MAD was defined as: A Duncan’s Multiple Range Test was used to test for differences among the MADs of the treatment groups. There was no significant difference among the groups in the responses on the pre-practice test. However, the ES groups did significantly better than either of the other two groups on the post-practice test. The Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 39 post-practice test results are presented in Table 2. These results indicate that the students using the ES decision aid made internal control assessments more like the expert after using that decision aid. Students using the more traditional decision aid (check list) did not significantly improve their performance as a result of practice. This was the case even though the questionnaire was designed to point the user to the factors identified as important in developing the ES. A post-test questionnaire surveyed the students attitudes toward the expert system decision aid, Overall the students viewed the expert system favorably and felt it aided in their understanding of internal control. They felt that the expert system decision aid was enjoyable, easy to use and made the decision process easier. They also indicate! that they did rely on the ES advice in making their decisions. CONCLUSIONS In this paper we have presented an efficient and cost- effective method for developing a classroom decision aid using an example based expert system shell. The necessary software investment is relatively small and “satisfactory experts are readily available though industry and public accounting contacts. Not only are students exposed to the latest concepts in decision support systems and fifth generation software, there is evidence that the use of an expert system is a positive option for reinforcing critical class concepts. Developments in Business Simulation & Experiential Exercises, Volume 15, 1988 40 APPENDIX II A Sample Scenario The internal auditor distributes the payroll on the first of every month as a control measure. The internal auditor regularly compares the amount of the payroll with the budgeted figure and investigates any significant differences. All employee complaints about their pay are handled by the internal auditor. During last year's audit, the evaluation of internal control over factory payroll was determined to be strong (i.e. above 50 on a scale of 0 to 100). The payroll clerk prepares the paychecks and the payroll register using the hours from the time cards and the current pay rate. The payroll clerk then posts the information to the individual earning records. Timekeeping checks the payroll register and prepares the payroll distribution voucher. Using the information from the job cards, timekeeping prepares the labor distribution summary. The general ledger clerk is responsible for comparing the payroll register and the labor distribution summary and reconciling any differences. The company does not use check signing machine and unsigned checks are not tightly controlled. Factory payroll is paid from the company’s only bank account. The cashier has responsibility for signing the checks after she thoroughly examines the payroll distribution voucher. The personnel department distributes the payroll checks. Any unclaimed payroll checks are retained by the personnel department. Employees manually record their starting and stopping times on time cards. The factory employees record the time of each job on jobcards which are approved by the supervisor. Any overtime worked is authorized by the supervisor. At the end of each work week, the total hours from the time cards are compared with total hours on the jobcards by timekeeping. Factory employees are hired by the personnel department which determines the appropriate pay rate. The personnel department sends notice of employment and the pay rate to the payroll clerk. All changes in pay rate are authorized by the personnel department. When factory employees terminate their employment, they must complete a form and submit it to the personnel. department which notifies the payroll clerk. REFERENCES Bailey, Andrew D. Jr., Gordon Leon Duke, James Gerlach, Chen-en Ko, Rayman Meservy, and Andrew B. Whinston, “TICOM and the Analysis of Internal Control,” The Accounting Review, April 1985, pp. 186-201. Dungan, C. and J.S. Chandler, “Auditor: A Microcomputer- Based Expert System To Support Auditors In The Field,” Expert Systems, 1985, 2(4), 210-221. Hansen, James V. and William F. Messier, Jr., “A Preliminary Investigation of EDP-XPERT,” Auditing: A Journal of Practice & Theory, Fall 1986, pp. 109- 123. Meservy, Rayman D., Andrew D. Baily, Jr. and Paul E. Johnson, “Internal Control Evaluation: A Computational Model of the Review Process, Auditing: A Journal of Practice and Theory, Fall 1986, pp. 44-74. Michaelsen, Robert H., “An Expert System For Tax Planning,” Expert Systems, 1984, 1(2), 149-167 Newquist, Harvey, III, “The Making of a Tax Expert,” AI Expert, March 1987, pp. 57-59. O’Leary, Daniel F., “Validation of Expert Systems-With Applications to Auditing and Accounting Expert Systems” Decision Sciences, Vol. 18, pp. 468-486. Olsen, Bruce, Bruce Pumplin, and Mickey Williamson, “The Getting of Wisdom: PC Expert System Shells,” Computer Language, March 1987, pp. 117-140. Steinbart, Paul J., “Materiality: A Case Study Using Expert Systems.” The Accounting Review, January 1987, pp. 97-116. Thompson, Bill, and Bev Thompson, “Creating Expert Systems From Examples,’ AI Expert, January 1987, pp. 21-26. ____________ and ____________ “Taxing the Expert System Shell,’ AI Expert, June 1987, pp. 23-29. Table of Contents Volume 15, 1988 The Role of Experiential Knowledge and Human Information Processing in Decision Making A Semantic Differential Instrument to Evaluate Experiential Teaching Methods A Comparison of Two Approaches to Management Skill-Building in an Organizational Behavior Course: A Replication Integrating Simulations: A Model for Business Policy Success Capstone Renaissance = Simulation + Interaction + DSS A Hybrid Method of Executing a Management Simulation: Combining the Best of Mainframes and Microcomputers Providing an Experiential Dimension to Cost/Managerial Accounting Courses Utilization of Computerized Tax Research Services in the Tax Research Curriculum Using and Expert System Based Decision Aid in Accounting Information Systems Event-Extended Entity-Relationship Diagrams for Understanding Simulation Model Structure and Function Multiple Objectives in the Development of the Gordon Macro Game A Comparative Study of Strategic Performance Factors in Actual and Simulated Business Environments An Empirical Investigation of Integrated Spatial-Proximity MCDM-Behavioral Problem Solving Technology Group Decision models Computer Simulation of Human Interaction The Role of Experiential Learning and Simulation in Teaching Management Skills Expert Systems - The New Business Simulation Tool Integrating Prolog into and Undergraduate Logistics Course Simulating Material Requirements Planning on Lotus 1-2-3 Innovation in Management Education: The Impact of the AACSB Experiential Learning in the International Environment Educational Testing with the Microcomputer A Simulation of Investment Analysis, Portfolio Management and Reporting Using Lotus 1-2-3 The Use of an Expert System to Develop Strategic Scenarios Two Exercises for Teaching about Motivation Sex Roles and the Good Manager A Form and Process for Nonconfidential Peer Evaluations Simulation and the Recalcitrant Student Employee Rights-Student Rights: A Classroom Exercise Computer Simulated Competition: An Alternative to Team Play Management Simulation The Relationship of Locus of Control and Vividness of Imagination Measures to Simulation Performance Formal Planning, Simulation Team Performance, and Satisfaction: A Replication Experimental Analysis of Magnitude and Source of Students' Inequitable Classroom Perceptions in Three Reward Conditions Strategy Design, Process and Implementation in a Stable/Complex Environment: An Exploratory Study Matching a Strategy Simulation to the Business Policy Literature: A Black Box Approach to Simulation Development An Evolutionary Classroom Experiential and Computer Simulation Model of a Corporate Strategic Planning System Collective Bargaining in the City of Elson: A Public Sector Experience Should Students Play Games in Labor Relations? Applying Cognitive Educational Objectives to Business Management Cases Grading as a Teaching and Feedback Mechanism: Involving Students in the Grading Process Teaching Controversies: A New Approach to Computer-Assisted Instruction and Simulation Packages Simulating Demand in and Independent-Across-Firm Management Game Advertising Response in the Gold and Pray Algorithm: A Critical Assessment A Model for Pricing Decisions in First Period Marketing Simulation Games Jog Your Right Brain: An Exercise for the Classroom and for Research Six Thinking Hats: An Exercise to Combat Confusion and Develop Thinking Skills Communicating in Context: A Simulation for Learning Business Communication A Simulated Consulting Service for the Compete Marketing Simulation Game Action Exams in the Consumer Behavior Class Using Focus Groups to Teach Problem Definition in Basic Marketing Research The Use of Journals in Management Simulations: A Literature Review and an ABSEL Response An Initial Step Towards Developing and Using an Expert System with a Business Simulation Self-Managed Learning: An Experiential Course Design Using the QWL Paradigm A Review of Current Developments in Experiential Learning Bring the Real World into the Classroom Assessing Student Performance on a Business Simulation Experience Minimizing Startup Anxiety: Case Studies of Simulation Experiences A Tale of Two Shepards Or Using Simulation in a Class Without Walls Enhancing Business Simulations Through the Utilization of Experiential Activities Involving Local Community Executives Simulation with Integrated Spreadsheets: The Design and Development of a Conversational Marketing Concepts Decision Game Introducing INMART: An International Marketing Simulation Using a Computer-Based Business Plan Assistant in Conjunction with a Marketing Simulation Game Overview of the ABSEL Guide to Experiential Learning and Simulation Learning