EVALUATION OF SIMULATION GAMES; A CRITICAL LOOK AT PAST EFFORTS AND FUTURE NEEDS Experiential Learning Enters the Eighties, Volume 7 1980 192 EVALUATION OF SIMULATION GAMES; A CRITICAL LOOK AT PAST EFFORTS AND FUTURE NEEDS A. Parasuraman, Texas A&N University ABSTRACT This paper discusses certain critical concerns regarding the usefulness of simulation games as educational tools. Based on an examination of relevant research studies that relate to the evaluation of simulation games, the paper argues that there is no conclusive evidence in support of the superiority of simulation games over traditional teaching methods. It finally outlines certain research guidelines that need to be followed by simulation game users, in order to mean- in a fully. and systematically, evaluate simulation games. INTRODUCTION The 1970’s have witnessed a significant growth in the use of simulation games in business courses. There is evidence of the formation of a cadre of college professors (many of whom regularly contribute to ABSEL Conferences) who appear to be loyal, regular users of simulation games. Though there is a large body of educators who are yet to get interested in simulation games, the individuals within the above-mentioned cadre of professors are well past the “interest” and “initial excitement” stages vis-à-vis the use of simulation games. Indeed, it is very encouraging to see that these professors have made, and are continuing to make, commendable efforts to evaluate the worth of simulation games as a pedagogical tool. The purpose of this paper is two-fold: (1) to critically examine past research efforts aimed at evaluating simulation games; and (2) to suggest some concrete proposals regarding the future evaluation of simulation games and their use in the classroom. PAST EFFORTS TO EVALUATE SIMULATION GAMES A review of published literature reveals three broad approaches that have been taken in the past to evaluate the worth of simulation games. The nature of each of these approaches, as well as the findings under each of them, are briefly described in this section. One approach to evaluating simulation games has been to teach several sections of the same course under different conditions (such as one section using a simulation game, another section using some cases, a third section using a straight lecture method, and so on), and to compare the performance of students across the various sections on a common test, administered to all sections (for example, see [3;6]). This avenue of e- valuation research has thus far produced no significant evidence to support the superiority of simulation gaining over other traditional teaching tools. In fact, a recent review done by Pierfy[7] of 22 studies that researched the effectiveness of simulation games vis-à-vis other teaching tools concludes that simulation games are no better than conventional teaching methods in fostering student learning. This is obviously disconcerting, especially in view of the increasing popularity of simulation games. Another avenue of research aimed at evaluating simulation games has focused on comparing students’ performance or grades in the simulation gaming part of a course with their performance on other “traditional” exams and assignments in the course. The results produced by this form of research have been mixed. Some studies have found no significant correlation between the two types of performance (see, for example, [1]) while others have found no significant difference between the two types of performance (see, for example, [5;8]) A third approach has been to ascertain the usefulness of simulation games by obtaining written feedback from students who took a course In which a simulation game was used. Typically, these feedback surveys contain questions relating to issues such as the following: Was the game interesting? ~Jas it useful? Was it theoretical or practical? Was it mentally challenging? Was this course more interesting than other traditional courses? Did it encourage interaction among class members? Should it be used in future courses? etc. It is this avenue of research that has by far been mostly in favor of simulation games. By and large, these types of surveys have produced pretty “positive” responses about courses containing simulation games (see, for example, [4;6]. The three approaches described above and their findings are critically examined in the next section. A CRITICAL EXAMINATION OF PAST EFFORTS Assessing the benefits of simulation gaming by comparing the performance of students exposed to simulation gaming with that of students exposed to other teaching tools may appear to be an “objective” evaluation method. However, as indicated earlier, this stream of research has produced no significant evidence in support of simulation gaming. This prompts one to come up with a couple of speculations, or hypotheses if you will: (1) Simulation games are at best only as good as “traditional” teaching techniques in making students learn the concepts to be taught in a course; and (2) Simulation games teach students “something” other than what traditional methods teach them -- “something” that cannot be measured by traditional tests. If the first hypothesis stated above is true, ardent supporters and users of simulation games will obviously have to ask themselves whether it is really worth the time, money, and effort to use simulation games, even though such games may be a lot of “fun”. If the second hypothesis is true, more work needs to be done to clearly determine what that “something” is which simulation games do teach and, more importantly, whether that “something” has any practical value to the students after they leave a course. The second approach to evaluating simulation games (1. e. comparing students’ grades or performance on simulation games with their performance on traditional segments of a course) is fraught with conceptual and methodological problems. These problems revolve around Experiential Learning Enters the Eighties, Volume 7 1980 193 the question of whether a student’s grade in a simulation game is a valid measure of anything at all. In other words, what does a high or a low grade based on performance in a simulation game indicate? Since most business games are played by groups of students, and are interactive in nature, it is hard to answer the above question with any degree of confidence. A detailed discussion of this issue can be found else- where [2]. Different studies using this research approach have led to inconsistent findings--perhaps involving thereby that simulation game grades may in act have been indicative of something unknown and different in each situation or study. So, unless research can clearly establish that students’ grades in simulation games are indicative of some unambiguous aspect(s) or process(es). this avenue of research will b fruit less in terms of eva1uating general benefits of simulation games. The overall, evidence based on the last approach described earlier (i.e. using surveys to get feedback from students) is quite encouraging at first glance, since it appears to be in favor of using simulation gaming as a teaching tool. Nevertheless, some careful examination and thought about these studies brings to mind some troubling question. The supportive evidence generated by these surveys is based on “self-reports” made by college students. Though such evidence is valuable to a certain extent, it is nevertheless “subjective” in nature. For instance, how qualified are these students to make an objective assessment of the benefits of a simulation game? In other words, is it possible that the overall positive response to simulation games may really he the result of the ‘novelty” and “fun” aspects (perhaps a “halo” effect) of participation in them? At the risk of appearing to be facetious, let me propose the following hypothetical (perhaps ridiculous, but hopefully thought-provoking) situation: Suppose there are two sections of a “boring” required course taught by the same instructor. Suppose the only difference between the two sections is that the instructor decides to include a simulation game in one section (Section I say) and a set of picnics, in lieu of the simulation game, in the other section (Section II). Whatever class time Section I spends on the simulation game is to be spent by Section II in going on picnics, or simply socializing and having a good time. In this hypothetical situation, if a common test is given to both sections at the conclusion of the course, there will likely be no significant difference in performance between the two sections (as supported by evidence available thus far). If a feedback survey is given to each section to assess the students’ reactions to each of the two alternative “teaching” tools the “picnic methodology” is likely to fare as well as, if not better than, the “simulation gaming methodology” on several of the typically used survey items such as the following: * “The course was interesting.” * “I would recommend the use of this in future courses.” * “This course was better than other courses I have had.” * “There was a lot of interaction among participants in this course.” etc… To be impartial, it should be stated that Section I may perhaps fare better than Section II on items such as the following: * “This methodology was helpful in understanding concepts well” * “This course was mentally challenging.” etc. However, the possibility of “positive” responses to such items showing up even in Section II, due to some sort of l!~1Il effect, cannot be entirely ruled out. Without belaboring this point any further, my position is that judging the worth of simulation games solely on the basis of feedback from college students is unwarranted and inadequate. SUGGESTIONS FOR FUTURE EVALUATION EFFORTS Performance on common tests taken by those exposed to simulation games and those exposed to other pedagogical tools have yet to show any advantage for simulation gaming. If simulation games do in fact teach something other than concepts measured by conventional tests, there is a pressing need to identify whatever knowledge simulation games do impart (or, are supposed to impart), and then to devise tests to measure the extent of acquisition of such knowledge by students. It is well known that the key “selling point” of simulation games is that they provide meaningful “hands on” experience for students not provided by traditional methods. A good way to measure the extent to which simulation games do in fact accomplish this objective would be to ask practioners in the real world to evaluate the usefulness of simulation games. Today business games are being increasingly used in executive development programs, management seminars, and the like, where the participants are those who already have at least a certain degree of real world experience. They will certainly be in a better, more qualified, position to evaluate the worth of simulation games in terms of the extent to which they are realistic, and the extent to which they help impart knowledge that is relevant for real world decision making. Yet, surprisingly, almost all the available evidence to date is based on the opinions of “traditional” college students, with little or no real world experience. It behooves us as simulation game enthusiasts to take immediate corrective action in this regard. Apropos of ascertaining the practical worth of business simulation games, let me “think aloud” a little bit. Would it not be beneficial if, every time a professor uses a simulation game in a program with practitioners as participants, he or she makes it a point to get feedback from them regarding the usefulness of the game? Perhaps all interested ABSEL members, as a body, :an: (1) develop a brief, but comprehensive questionnaire than can be standardized, and used for this purpose by all members; and (2) set up a central “data base” for the collection and analysis of such feedback information obtained from practioners. Attempting to establish a data base along the lines suggested above may sound like too idealistic an undertaking. However, I feel that, with a little effort from all of us interested in simulation gaming, such a system can be made operational, and can be quite valuable. Over a period of time a large enough data base can be created to evaluate: (1) the worth of simulation gaming, in general; (2) the worth of individual simulation games used repeatedly in different programs: and (3) the worth of possible modifications that can he made in a game to improve it. Experiential Learning Enters the Eighties, Volume 7 1980 194 SUMMARY AND CONCLUSIONS It is encouraging to see that conscious efforts have been made to evaluate simulation games. However, to date there is no conclusive evidence to clearly show that simulation gaming is superior to other techniques in terms of teaching something that is of any practical consequence or relevance. Hence a more coordinated effort on the part of simulation game users is urgently needed to establish the real value, if any, of simulation gaming. This is especially important since the use of simulation gaming requires much more time, effort, and money compared to traditional methods. The 70’s have seen a growth in the popularity and acceptance of this novel teaching tool. Let us hope that the 80’s will witness an organized effort to clearly establish the practical benefit and relevance of this novelty. REFERENCES [1] Badgett, Tom F., Daniel C. Brenenstuhl, and William J. Marshall, ‘An Analysis of Performance in Simulation Games Compared to Performance on Structured Course Criteria: A Case Study,” Proceedings, Fifth ABSEL conference, Denver, 1978, pp. 32-38. [2] Butler, Richard J., and A. Parasuraman, “Integrating Business Game Performance with the Grading Process,” Proceedings, Midwest AIDS Conference, Indianapolis, 1975, pp. 93-97. [3] Catalanello, Ralph F., and Daniel C. Brenenstuhl, “An Assessment of the Effect of Experiential, Simulation and Discussion Pedagologies used in Laboratory sections of an Introductory Management Course,” Proceedings, Fourth ABSEL Conference, Wichita, 1977, pp. 51-58. [4] Chisholm, Thomas A., Parameswar Krishnakumar, and James P. Clay, “An Exploratory Investigation of Student Perceptions of Computer Simulation as an Educational Tool, “Proceedings, Fifth ABSEL Conference, Denver, 1978, pp. 330-336. [5] Estes, James E .,“Research on the Effectiveness of using a Computerized Simulation in the Basic Management course,” Proceedings, Sixth ABSEL Conference, New Orleans, 1979, pp. 225-228. [6] Mancuso, Louis C., “A Comparison of Lecture-Case Study and Lecture Computer Simulation Teaching Methodologies in Teaching Minority Students Basic Marketing,” Proceedings, Second ABSEL Conference, Bloomington, 1975, pp. 339-346 [7] Pierfy, David A., “Comparative Simulation Game Research -- Stumbling Blocks and Stepping Stones,” Simulation and Games, Vol. 8, No. 2, June 1977, pp. 255-268. [8] Wolfe, Joseph, “Correlations Between Academic Achievement, Aptitude, and Business Game Performance,” Proceedings, Fifth ABSEL Conference, Denver, 1979, pp. 316-324. Table of Contents Volume 7, 1980 Symbol Recognition and Correlation for Evaluating Decision Making in Computer Aided Business Simulations Polanal: An Experiential Approach to Decision Support The Use of Time Contracts in Formal Education Indexing Simulation Model Response for Gaming Flexibility Moving Toward A Theory of the Use of Simulation Games and Experiential Exercises Use of Simulation Administration to Achieve Pedagogical Objectives Experiential Learning on the Job - A Business Internship Program Toward the Ultimate Experiential Exercise A Learning Through Managing Program Workshop Using Experiential Materials in Industry Training Sponsored Experiential Learning - An Opportunity Problems and Pitfalls of Externally Sponsored Field Research Projects Viewed form an Experiential Learning Perspective A Modular Approach to Experiential Learning: Classroom & Consulting Using Simulation & Experiential Learning In Industrial Settings Terminations: An Experiential Review Agenda Items -- Board of Supervisors' Meeting - Town of Jori The Objective-Setting Interview in MBO: An Experiential Approach The All-Star World Series Team Exercise: an Experiential Learning Exercise Dealing with Various Organizational Behavior Issues Progress Report on global, A Rich Multinational Gaming Environment Computer Simulation: A Tool to Teach Queuing Theory New Technology for Business Games Technological Frontiers in Computer Simulations for Business Education Simulating the Product Life Cycle on Interactive Terminals On Compensatory Demand Functions in Marketing Simulations The Use of Games at Different Levels in a Single Marketing Course to Increase Game Participation Sun Airlines: A Heterogeneous Consumer demand exercise Incorporating a Group Selection Test An Organization Development Approach to Teaching Organization Behavior CBID: Cognitive, Behavioral, and Interpersonal Development - A Skill Development/Social Learning Approach to Management Development Using a Live Case Via Video Tape A Town and gown Approach Development of Student Generated Cases Using Computerized Text Editing and Database Technology Interdisciplinary Approached to Problems in Utilizing Experiential Techniques To Use or Not To Use Experiential Techniques, That's is the Question Forming Participant Teams in Simulation Gaming The Problems of Motivating Students and Clients in Live-Case Projects Problems of Teaching Leadership Skills Through Experiential Techniques Conflict Style Measurement: Antecedent to Change - A Proposal for an Experiential Exercise Demonstration Fundamentals of Simulation for Newcomers An M.B.A. Orientation Simulation for Managing Time and the Areas of One's Life SimNet Workshop: The International Simulation Network Demonstrates Three New Business Games The Use of a Simulation Model in the Planning and Evaluation of Commercial Bank Operations Probability Assessment and Performance in Business Game Simulations WageSim: A Wage and Salary Administration Simulation Grading as a Teaching and Feedback Mechanism: Modifying Student's Self-Perceptions of Performance Can Business Games Effectively Teach Business Concepts? Development of Multiple Value Orientations in Conflict Resolution Behavior: An Experiential Teaching Paradigm in Labor-Management Relations Course Designing a Competency-Based Peer Assessment Scale for the Evaluation fo Teaching in Higher Education A Method for Evaluating Information for the Equipment Replacement Decision: An Application of Monte Carlo Simulation Simulation: A Method of Appraising Communication Networks in Managerial Decision Making An Evaluation of In-Class Student Involvement Evaluation of the SBI Program from an Experiential Viewpoint: Focus on the Student Differential Predictors of Academic Performance for White and Non-White Samples The Manager's Dilemma: An Unobtrusive Measure of the managerial Sex-Role Stereotype Are Computer Simulations Sexist? The Effect of Group Size on Attitudes Toward the Simulation Associations Between Individual Cognitive Processing Variables and Business Game Performance and Play Students' Perceptions on Managerial Functions After Exposure to Either the Case Method or a Simulation What Business Students Learn from Finance Simulations Attitude Toward Experiential Exercises, The Student-Teacher Relationship, Student Psychological Types, and Performance An Example of How to Design a Research-Based and Classroom-Effective Organizational behavior Exercise Some Issues in Game Design Untested Hypotheses: An Approach to Experiential Learning Evaluation of Simulation Games: A Critical Look at Past Efforts and Future Needs Is Self-Perception Predictable? - Some Laboratory Results An Exercise in Conflict-Handling Behavior The Relationship Between Group Size and the Learning Curve Effect in a Gaming Environment Learning About Organizational Management Through Organizational Management: Closing the Gap Strategies, Managerial Approaches, and Decision Making in a General Management Simulation A Comparison and Evaluation of Similar and Dissimilar Group Scenarios Generated Using Manual Simulation Games Weaknesses of Research Methods in Experiential and Simulation Studies