USE AND EFFECTIVENESS OF AN ANALOGY-BASED EXPERT SYSTEM Developments In Business Simulation & Experiential Exercises, Volume 17, 1990 11 USE AND EFFECTIVENESS OF AN ANALOGY-BASED EXPERT SYSTEM Richard F. Barton, Texas Tech University T. M. Rajkumar, Miami University Jon Randolph, Texas Tech University ABSTRACT An expert system consisting of fast, pertinent retrieval of historical decisions and results was provided to business simulation game players. Use of the expert system improved player performance on the game grading criterion. Expert system usage was greatest at the outset of game play and declined to little or no usage near the end. It is inferred that the expert system aided more in the early strategic phase of game play than later when implementation and operating considerations dominate. Larger teams tended to use the expert system more. INTRODUCTION Recently there has been considerable interest in the development of expert systems for business simulation gaming (Sackson and Varanelli, 1988; Varanelli et al., 1987; Gautschi and Prasad, 1988; Rashid, Cannon, and Morgan 1988) Though initial descriptions of these kinds of expert systems exist, no empirical study has been conducted either on the usage or the effectiveness of these systems in a simulation gaming setting. Expert systems could potentially aid in strategy formulation, understanding of problem environments and increased decision-making effectiveness as reflected by player performance. This study reports an initial investigation of the use and effectiveness of an analogy-based expert system by player teams in a management simulation game. AN ANALOGY-BASED EXPERT SYSTEM Development of a reasoning by analogy (RBA) system for a policy course simulation game was done in early 1988. This system was described in an earlier paper (Rajkumar and Barton, 1989). Briefly, in this system two semesters of past game plays serve as a knowledge base. The complete listings of all firms in all industries were included regardless of level of performance. Player users can query this knowledge base and retrieve a company report for a past firm for a particular decision period. This report then might serve as an analogy for their current gaming situation. More than one potential analogy may be retrieved. Each analogy may serve as a possible decision solution for the player team’s pending decision. The simulation game for which the RBA expert system was implemented (Barton, 1978) was embedded in a decision support system (DSS) that enabled teams to “simulate the simulation” (Barton, 1981) before finalizing an official decision for the current gaming cycle. Teams could thus try out as many decisions as they could think of for the amount of time they wished to invest. The RBA expert system provided possible decisions from past game plays for such trial simulations. The RBA expert system operated through a separate computer program so that teams used two terminals during their decision sessions. There were many query criteria, including compound criteria, so that the search for analogous firms was easy and fast. Retrieved analogies were displayed on the terminal screen in exactly the same form as the DSS and official printed output. No facilities for printing the analogies were provided. All expert systems inherently have the following characteristics: (a) have an expert level of knowledge of the domain of expertise, (b) generate solutions to problems in the domain, and (c) explain lines of reasoning. In this RBA expert system, the knowledge of the domain is achieved in two fashions, 1) the system contains all the rules and knowledge of the game because the computer game model itself is incorporated into the expert system; and 2) the applicability of the solutions in particular environmental situations is achieved by using the experience of past winning teams as the experts. The decision set of the retrieved analogical firm (winning team or not) serves as a solution to try in the users current gaming situation. The third criteria--the ability to explain lines of reasoning--is viewed by many researchers to be a critical feature of expert systems. However, this RBA expert system was meant for research and pedagogical purposes and as such was designed to stimulate the students creativity in reasoning, so explanations of lines of reasoning were not provided. However, details of competitors’ decisions and reports normally hidden during regular: play were made available. In addition, on-line help facilities were provided. EXPERIENCE WITH THE RBA EXPERT SYSTEM The impact of this RBA expert system was evaluated by concentrating on two questions: I) Was the expert system used? and 2) Did the performance of decision making improve when the expert system was used? This approach has been used by others to evaluate the effectiveness of both DSS (Sharda, Barr and McDonnell, 1988) and expert systems (Fordyce, 1987) - Gaming Environment The expert system arid DSS were used by students enrolled in a capstone business policy course. As part of the course, students were divided into teams, and each team managed a firm in a simulated industry. Two industries were simulated during the semester. The first simulation (called a mini-play) was of the ‘‘Breakfast Cereals’’ industry and was played over a period of four weeks. A practice decision and three official decisions were made by players in the four-week period. During the mini-play, players were trained in game parameters and rules, and in both the DSS and the expert system. On completion of the mini-play, a more complex “Microcomputer Industry” was simulated (called a maxi- play) for a period of eight weeks, with one practice week and seven official decisions. The experimental setup used by the students is shown in Figure 1. When a user wanted to see an analogy, he entered a query and a computerized pattern-matcher searched the knowledge base and retrieved analogies. One or more analogies may then be used by player-teams to formulate a decision set to be tried on the DSS in the team’s current environment. There were three patterns of usage evident in this set up: 1) beginning game pattern, 2) regular pattern, and 3)-end pattern. These are discussed in the next subsection. Usage Pattern of the Expert System At the beginning of the game, there is a large amount of unstructured data for the teams because of the new industry environment and the unknown plans of competitors. This lack of structure resulted in extensive use of analogical reasoning by the players as evidenced by numerous searches on the firms in the knowledge base that had won past plays. The decisions and reports of these firms were studied for an entire play on a period-by-period basis, especially the beginning period decisions. The strategies used by these analogical firms (winners) were compared and discussed before some of these strategies were tried on the DSS. Thus, analogical reasoning shaped initial strategies and helped structure the uncertainty in the Developments In Business Simulation & Experiential Exercises, Volume 17, 1990 12 environment. This beginning pattern might be thought of as a burst of queries. (Data reported in later paragraph.) As play progressed into the second and third weeks, a more regular pattern was established. Both terminals were used simultaneously; each team had one member on the DSS and another member on the expert system. The person on the DSS entered the decisions and performed what-if analyses. The expert system was used to find an analogy appropriate for the current period. The player using the expert system would look at the analogical firm and relay the decision element values used by the analogical firm to team members. Discussion normally ensued, decisions were adjusted to fit the team’s current situation, and then tried on the DSS. After seeing the results on the DSS screen, the decision values were either further adjusted based on trial results, or another analogy was sought (as the expert system generally found more than one analogical firm) and the process repeated. This pattern may be thought of as use of the RBA expert system with some regularity. As play progressed further into the fourth through seventh weeks, the use of the expert system declined while regular usage of the DSS continued. This end pattern could have been due to the fact that by then team, were familiar with their environments and the strategies of each team had stabilized. Consequently, there was less uncertainty and less need for creativity. The number of analogies retrieved by the players followed the declining usage pattern of the expert system. In the first decision period an average of 12 analogical firms and 3 competing firms were seen by each team. This dropped to 6 firms and 2 competing firms for the second decision period. Decision period 3 and 4 resulted in 3 analogical firms and 2 competing firms being retrieved. Decision period 5 averaged I analogical firm being retrieved. Decision periods 6 through 7 resulted in very few analogical firms being seen by game players. Effectiveness of Decision Making Improvement in gaming performance was tested by comparing whether the team., made better decisions when the expert system was available than when it was not. The criterion for performance compared was the same as used for team grading: simulated stock market quotation. For the maxi-play the criterion provided two grades. One was the market quote achieved at the end of the maxi-play. The other was an average of the market quote for the first six decisions. The expert system has been used for two semesters (fall 1988 and spring 1989) by sixty teams. This one-year of experience was compared to two prior years experience with 60 teams when the RBA expert system was not available. These past teams faced the identical gaming environment as the expert system teams, i.e., same game version, no parameter changes, three or four players per team, and three or four firms per industry. The following hypotheses were tested: Hi: The teams that used the expert system will not achieve a higher level of decision making as evidenced by average market quote. The teams that used the expert system will not achieve a higher level of decision making as evidenced by ending market quote. The results of the tests are provided in Table I. They indicate that Hypothesis 1 can be rejected at the probability level of .054, and Hypotheses 2 can be rejected at the probability level of .01. Rejection of both hypotheses indicates that the expert system makes a significant difference in decision making effectiveness as evidenced by market quote performance for this kind of gaming environment. Further validation was obtained through oral feedback. Players using the system stated that the system shaped their initial game strategies. Two of the winning teams also attributed their winning to the use of the expert system. The effects of two other variables--team size and quality of players on expert system use, DSS use, time required to make the decision and performance were studied. These are discussed in the next two subsections. Team Size Team size has been found to be a determinant of performance in simulation gaming environments (Wolfe and Chacko, 1983) . A3 membership increases the number of potential information exchanges increases. Increased team size can bring in additional cognitive capability, technical skills and energies to proceeds further information and alternatives (Harrison, 1975) Hence, larger teams may perform better than teams of smaller size. In addition, team size may affect the use of expert systems and DSS positively because a larger team can proceeds more information. However, increased team size would lead to a longer time to make a decision as it is more difficult to achieve consensus in larger teams (Wolfe and Chacko, 1983) Usage of the expert system and DSS was measured as the number of queries run on the expert system and the number of what-if trials run on the DSS during the initial two week period before observed usage started to decline. The measures were unobtrusive and tracked on the computer. The time taken to arrive at a decision was measured as the time spent logged on the DSS terminal. Size was either 3 or 4 (the number of players on a team). There were 14 3- member teams and 28 4-member teams. Performance was operationalized as the single grade achieved at the end of the game. The following hypotheses were tested: H3: The teams that had 4 members will not achieve a higher level of decision making as evidenced by the final grade. H4: The teams that had 4 members will not use the expert system to a greater extent as evidenced by the number of queries. H5: The teams that had 4 members will not use the DSS to a greater extent as evidenced by the number of what-if trials. H6: The teams that had 4 members will not take longer to arrive at a decision as evidenced by the computer logon tune. The results are reproduced in Table 2. Due to a change in the system after the fall semester, discussed later, only the spring semester data are Developments In Business Simulation & Experiential Exercises, Volume 17, 1990 13 reported. As can be seen, seen, team size had a significant effect on expert system use. No significant effect on the DSS use was found, which may be because the DSS use in the experimental environment was almost mandatory while expert system use was more voluntary in nature. Size relative to performance and the time taken to arrive at a decision were not significant. The lack of effect of size on performance has also been evidenced in other studies (Norris and Niebuhr, 1980) Quality of Players Previous studies using students in simulated talks have reported difference in performance due to academic background and ability. For example, Seginer (1980) found a relationship between academic ability and team performance. Hence, teams with higher average grade point average for past courses (GPA) would perform better than teams with lower GPA. Lucas (1975) has stated that background factors such as education affect the use of information systems. Because of education, users may apply analytic techniques to the output and be able to reason out relationships, and hence use more output. In contrast, teams with lower GPA would use the expert system and DSS in a briefer more brute force fashion for identifying potential decisions. In addition, teams having higher GPA way have more motivation to maintain GPA’s and would welcome additional information and hence use an expert system and a DSS more than teams with lower GPA’s. On the other hand, teams with higher GPA’s may also be more efficient in managing their tune, and hence would arrive at decisions faster than others. The average GPA of the teams were calculated. Team., were classified as high GPA team if the average GPA exceeded 2.70, which is a common cut-off level for entry into upper division programs for these players; otherwise a team was classified as low GPA team. There were 22 high GPA teams and 20 low GPA team.,. The following hypotheses were tested: H7: High GPA teams will not achieve a higher level of decision making performance as evidenced by the final game grade. H8: High GPA teams will not use the expert system to a greater extent as evidenced by the number of queries. H9: High GPA teams will not use the DSS to a greater extent as evidenced by the number of what-if trials. H10. High GPA team., will not take longer to arrive at a decision as evidenced by the computer logon time. The results are reproduced in Table 3, which, again, shows averages only for the first two periods for the spring semester. of the effects of GPA on the four variables considered, the time taken to arrive at the decision was the only variable to be significantly affected by GPA. The longer time taken by teams with higher GPA may have been caused by more motivation to invest time to maintain their GPA’s than lower GPA teams. However, if so, hopes did not come true since GPA did not affect performance. GPA also did not significantly affect the use of the DSS and the expert system. All this may be due to the fact that GPA alone does not determine the background of the teams. Other factors such as work experience may affect how they use the expert system and DSS. The lack of effect of GPA on performance is not surprising as other studies (Norris and Niebuhr, 1980; Gosenpud, Milton and Larson, 1985) have reported similar results. CONCLUSIONS Experience with this expert system shows that it is of value and aids in making better decisions on the part of the players (Table 1), but only in environments of uncertainty and in the strategic phase of game play. This may be due to the fact that at the moment of the first decision, players are able to generate 100% analogies to their environment. Then, as play progresses the interpretation of the analogical firms may require processing of larger information by the players as they now try to see whether the analogical firm used similar strategies in the past as their firm, or that the current positions of competitors are similar, to make sure that it i3 an appropriate analogy. However, such situation rarely occurs after the first few periods, arid usage of the expert system drops. (There was no observed increased use of other forms of analysis to account for the decline in usage of the expert system.) Team size affected only expert system use (Table 2) This implies that particular attention needs to be oriented on this aspect when the teams &re formed as expert system use significantly affects performance of the team. It has also been shown that teams with higher GPA tend to make their decisions slower, but not necessarily better as indicated by the lack of significance of GPA on performance (Table 3) Other observations have to do with the novel ways in which players used the expert system. One teem calculated the average decision made by all firms in the knowledge base, then took the averages as predictors of current competitors’ decisions. In the first semester, players could not easily and rapidly trace a single firm through all periods of game play. The design of the system was changed for the second semester to accomplish this. The nature of the queries then shifted more to the winning ending criterion followed by tracing the retrieved analogy back to its initial period decisions. This implies that the future design of such systems could eliminate many of the complex query features provided in this system, which would speed search processing. It was also noticed that of the 50 firms in the knowledge base about 5 of the firms were being retrieved most often by the game players. These firms had some of the highest values on the game winning criterion. Additionally, some of these firms were retrieved more than once each week, and throughout the use of the expert system one or two firms were consistently seen by many of the teams to identify the strategy used by these winning firms. Developments In Business Simulation & Experiential Exercises, Volume 17, 1990 14 REFERENCES Barton, R.F. (1978), The. IMAGINIT Management Game, Lubbock, TX: Active Learning. Barton, R.F. (1981), “Simulating the Simulation for Enhanced Player Rationality”, Proc. Eighth ABSEL Conference, 269-272. Fordyce, K.J. (1987), “Looking at worksheet modeling through expert system eyes”, in Silverman, B.G. (ed.), Expert Systems For Business, 246-285, Reading, MA: Addison-Wesley. Gautschi, T.F., Prasad, M. (1988), “An initial step towards developing and using an expert-system with a business simulation”, Proc. 15th ABSEL Conference, 225-230. Gosenpud, J., Milton, C., Larson, A. (1985), “Predicting performance over the course of the simulation”, Proc. 12th ABSEL Conference, 5-10. Harrison, E.F. (1975), The Managerial Decision-Making Process, Boston: Houghton-Mifflin. Lucas, H.C. (1975), “Performance and the use of an information system”, Management Science, Vol. 20, No. 8, 908-919. Norris, DR., Niebuhr, R.E. (1980), Group variables and gaming success, Simulation and Games, Vol. 11, No. 3, 301-313. Rajkumar, T.M., Barton, R.F. (1989), “A knowledge based system to support reasoning by analogy for business simulation gaming”, Proc. 16th ABSEL Conference, 97-102. pp. 97-102. Rashid A., Cannon, H.M., Morgan, F.W. (1988), “A model for pricing decisions in ‘First period’ marketing simulation games”, Proc. 15th ABSEL Conference, 192-196. Sackson, M., Varanelli, A. (1988). “The Use of an Expert System to Develop Strategic Scenarios”, Proc. 15th ABSEL Conference 106-112. Seginer, R. (1980), ‘Game Ability and Academic Ability”, Simulation & Games, Vol. 11, Dec., 403-421. Sharda, R., Barr, S.H., McDonnell, J.C. (1988), “Decision support system effectiveness: A review and an empirical test”, Management Science. Vol. 34, No. 2, Feb., 139-159. Varanelli, A., Sackson, M., Cronin, D., Dulberg, C.L. (1987), “The Use of Expert Systems to Develop Strategic Scenarios; An Experiment Using a Simulated Marketing Environment”, Proc. 14th ABSEL Conference, 208-213. Wolfe, J., Chacko. TA. (1983), “Team size effects on business game performance and Decision-Making Behavior’s”, Decision Sciences, Vol. 14, No. 1, 121- 133. Table of Contents Volume 17, 1990 The Impact of Decision Support Systems on the Effectiveness of Small Group Decisions - Revisited The Relationship Between Financial Performance and Other Measures of Learning on a Simulation Exercise Use and Effectiveness of an Analogy-Based Expert System Suggestions for Computerized Business Authors Dealing with Power: An Experiential Exercise Using Movie and Personal Diary Analysis Techniques A Model for Developing Student Skills and Assessing Outcomes Through Outdoor Training Computer-Aided Exercises Versus Workbook Exercises as Learning Facilitator in the Principles of Marketing Course An Exposition of Guilford's Si Model as a Means of Diagnosing and Generating Pedagogical Strategies in Collegiate Business Education Formal Planning and Simulation Team Performance: A Cross Sectional Approach Cases: Real Organizations in Real Time in the Classroom An Empirical Investigation of the Internal Validity of Marketing Simulation Game An Empirical Evaluation of the Pedagogical Value of Playing a Simulation Game in a Principles of Marketing Course Factors Affecting Effective Teaching of Strategic Planning: Some Preliminary Evidence An Experiential Exercise for Learning About the Relationship Between Organizational Form & the Project Management Process Accounting Communication Skills can be Taught in the Auditing Course Modeling Cost Functions in Computerized Business Simulation: An Application of Duality Theory and Sheppard's Lemma A Life Cycle Analysis of Decision Making for a Strategic Management Team What's the Problem? A Dynamic Model for Teaching Problem Solving Skills Experientially International Currency Fluctuations: Money$im, A Simulation Superstores: A Specialized Retailing Simulation Within a Specialized Marketing Curriculum Factors Affecting Student Perceptions of Learning in a Business Policy Game VC + EL = VL The Name Game: An Experiential Exercise in Intergroup Relations The Effects of Experiential Accounting Work Experience on Student Performance in Intermediate Accounting Courses The Results of Using the Experiential Activity Group Performance Evaluation in a Business Policy Setting Using a Legal Database to Describe the Legal Environment of Marketing (and Business) Matching Environmental Uncertainty and Organizational Configuration An Instructional Computer Simulation of Tampering in QC Executive Evaluation of Student Learning in the Looking Glass Simulation Group Personality Composition and Total Enterprise Simulation Performance An Expert System for Selecting Analytical Techniques for Analyzing Marketing Research Data Effects of Cognitive Styles on Responses in an In-Basket Simulation A Psychometric Analysis of Kolb's Revised Learning Style-Inventory Selecting and Developing Experiential Exercises Using Movies Application of a Real-World Strategic Management Model in the Classroom Demand Equations which Include Product Attributes Consumption as the Objective in Computer-Scored Total Enterprise Simulations The Effects of Decision Format and Evaluation on Simulation Performance, Decision Time, and Team Cohesion The Effects of Computer Related Assignments on Student Performance in Business Administration The Money Game: A Dynamic Simulation Including Random Shocks for Money and Banking Courses Methods for Evaluating Performance on Business Simulations: A Survey The Effects of Synergogy on the Policy Course: Significant Improvements in Student Learning and Teacher Evaluation Conditions and Outcomes of Trust in a Two-Person Bargaining Exercise Bankgame Enhancing Computer Business Simulation with the Use of VGA Graphics An Experiential Approach to Entrepreneurship An Advanced Simulation Method (ASM) for Multiple Objective Problems The Influence of Experiential learning Techniques on Student Recognition of Non-Primary Learning Styles Negotiating Mergers and Acquisitions: A Cocktail Napkin Approach Identification of Unintended Effects in Experiential Laboratory Exercises An Experimental Comparison of Paper and Pencil and Computer Aided Decision Support Tools Porting a Simulation from the IBM World to the Macintosh World A Transaction Cost Analysis of Experiential Learning The Development of Experiential Exercises for Courses in Entrepreneurship and Small Business Management The Assessment Center as and Experiential Classroom Exercise Pricing Strategy Algorithms for Playing Business Simulations Organizational Structures for International Operations: An Experiential Activity Simulation Emphasis in the Business School Capstone Course An Integrated Approach to Computerizing the Business Curriculum Cross-Cultural Business Negotiations Exercise Organizational Socialization and Gender Differences in Students at Work Understanding Student Work Experience: A Content-Analytic Approach Introducing Executive MBA Programs with Management Games An Analysis of Improvement in Business Decision Outcome with Sequential Use of Two Simulation Games Teaching Business Policy Utilizing Mass Lecture and Individual Case Labs Potholes Along the Road to Evaluating Learning Outcomes: The Case of Outdoor Management Training Experiential Learning for Interior Design Students: Using CADD, Lotus 1-2-3, and Wordperfect Cognitive Learning Using a Computer-Based, Qualitative Interactive Business Simulation Sex Discrimination: Does the Woman get the Job or Does the Best Man Win? A Search for Visual Aids to Support Experiential Learning Through the 1990's An Experimental Analysis fo the Effectiveness of Student Role-Playing in Sales Training How to Have Students Learn from their Term Projects Self-Evaluation Exercise (SEE): An Assessment of Class Contribution A Study of the Influence of Team Formation on Attitudes and Performance in Management Games A Hardware Based CIM Simulation Laboratory Model Test Substance Abuse in Organizations Micro Computer Training Models Teaching Forecasting, Cash Budgeting and Inventory Model Building Using SBTools A Comparison of the Effects of Experiential Learning Activities and Traditional Lecture Classes An Adjunct Writing Instruction Assistant, The Computer; With an Illustration Classroom Software for ABC Analysis Utilizing Information Processing Technology to Enhance the Business Policy Simulation Experience Progressive Cases Realistic Job Previews Vs. Traditional Job Previews: Experiencing the Differences and Understanding the Consequences Investment Analysis Using the Pragmatic Multiplier Approach A Computer Simulation Interface for Competitive and Firm Analysis Self-Assessment of Ethical Decision Making Predispositions Preparing Managers for Overseas Assignments An Inquiry into Japanese Marketing: Workshop on Teaching Japanese Marketing The Performance Appraisal Feedback Interview: A Role Play for Human Resources Management Teaching Counselor Selling techniques Using Experiential Techniques to Teach International Topics International Management Simulation Gaming: Current Status and Future Developments A Realism Comparison of Simulation Technologies/Methodologies A Time-Efficient Game to Illustrate Concepts Taught in Management Courses and Management Development Programs Teaching the Management of Technology The Concept of Face and the Applicability of Experiential Exercises in an Oriental Culture's