EDUCATIONAL EFFECTIVENESS OF BUSINESS SIMULATION GAMING: A COMPARATIVE STUDY OF STUDENT AND PRACTITIONER PERSPECTIVES Development In Business Simulation & Experiential Exercises, Volume 18, 1991 53 EDUCATIONAL EFFECTIVENESS OF BUSINESS SIMULATION GAMING: A COMPARATIVE STUDY OF STUDENT AND PRACTITIONER PERSPECTIVES Masoud Hemmasi Lee A. Graf Illinois State University ABSTRACT The purpose of this study is to clarify some of the issues surrounding the educational value of business simulation gaming. It examines and compares student and practitioner perceptions of simulation effectiveness in enhancing participants’ appreciation of, and skills in, such areas as interpersonal relationships, theory application to real-world problems, planning, decision making/problem solving, conceptualization/integration of business functional issues, and environmental analysis. INTRODUCTION More than three decades have passed since in 1956 the first practical business game was introduced by the American Management Association. Since that time, the number of available collegiate-level business games has mushroomed. By 1980, the field’s major sourcebook identified and described 228 different business games available for use in education and training (Horn & Cleaves, 1980). Surveys report that simulation game usage among AACSB member schools increased from 71 percent of respondents in 1962 to over 95 percent in 1987 (Faria, 1987). The growth of simulation game usage has provided the impetus for the emerging body of research literature concerning this instructional pedagogy. To date, researchers have examined such related issues as organizational learning (Cangelosi & Dill, 1965); influences on learning in business games (Chisholm, Krishnakumar, & Clay, 1980); the effect of information (Biggs & Greenlaw, 1976); the effects of advisors (Starbuck & Kobrow, 1966); real-life applicability of learning through simulations (Wolfe & Roberts, 1986); the impact of game complexity on learning (Wolfe, 1978; Butler, Pray, & Strang, 1979); and performance and behaviors of undergraduate and graduate business students in simulation games (Hemmasi, Graf, & Kellogg, 1989). The types of managerial skills (e.g., problem solving/decision making, teamwork/interpersonal) that students are expected to learn through this instructional tool also have been a subject of continuing inquiry in the above stream of research. Keys (1976), and Wolfe (1985) provide excellent reviews of the more rigorous studies on the above subject. These comprehensive reviews, however, point to the inconclusive and contradictory findings of such investigations. In regard to the issue of skills transfer, Partridge and Sculli (1982), for example, found simulation gaming to be most effective in developing decision making and information processing skills, but ineffective in developing peer and leadership skills. Conversely, Roderick, Wilterding, and Eldredge, (1979) reported simulations to be a highly effective means of improving interpersonal skills, while Miles, Biggs, and Schubert, (1986) found simulations to be less effective than other methods in improving problem solving skills. Such clear inconsistencies in the literature can be attributed, in part, to methodological and research design imperfections (Stone, 1982). Some studies, for example, involved “comparing simulations to other pedagogies in the abstract, mixed diverse groups such as MBAs and undergraduates, and did not use statistical analysis” (Miles, et al., 1986: 11). Others have based their conclusions solely on surveys of business school deans (e.g., Faria, 1987). Such considerations have prompted critics to call for use of standardized questionnaires and reporting procedures, more rigorous research designs and methods of analysis, and collection of data from both students and practitioners (Parasuraman, 1981; Wolfe, 1985; Miles et al., 1986). The research reported here is a step in that direction. RESEARCH QUESTIONS AND OBJECTIVES Specifically, the present study was designed to overcome the above mentioned shortcomings. It examines and compares student and practitioner perceptions of the educational effectiveness of simulation gaming in enhancing participants’ appreciation of, and skills in, such areas as teamwork/interpersonal relationships, theory application to real-world situations, planning, problem solving/decision making, conceptualization and integration, and environmental analysis. The student and practitioner perspectives on the subject were obtained through a survey of both business students participating in an all enterprise simulation game, as well as business alumni (hereafter called practitioners) who, prior to employment, had been exposed to the same simulation experience as students. We expect that the years of experience that practitioners have gained since graduation will put them in a unique position to retrospectively, and perhaps more realistically, judge the true educational value of their collegiate simulation experience. In addition, as was mentioned earlier, it is conceivable that the inconsistencies in the existing literature are, at least in part, attributable to the different types of research subjects (i.e., students, practitioners, deans) used by different investigators in previous studies. By simultaneously studying and comparing the student and practitioner data in this study, we may Development In Business Simulation & Experiential Exercises, Volume 18, 1991 54 have the unique opportunity both to examine the plausibility of the aforementioned contention and, then, perhaps explain and put to rest some of the controversy that surrounds the subject. Finally, this study will ascertain practitioner/student perceptions concerning the extent to which each of the skills and concepts learned through simulation gaming has been/will later be helpful in their performance on the job. The research propositions to be tested are presented below. Notice that because of the exploratory nature of the study and the inconclusiveness of evidence in the existing literature, these propositions are stated in a non--directional form. Prop 1. There is a significant difference between student and practitioner perceptions regarding educational effectiveness of business simulation gaming in teaching various managerial skills. Prop 2. There is a significant difference between student and practitioner perceptions regarding the importance of various managerial skills to job performance. METHODOLOGY Sample and Data Collection The sample for this study is composed of two groups-l46 senior business students at a large midwestern university and 124 of the university’s business alumni. Student data were collected (over two semesters) at the conclusion of the semester through a standardized questionnaire from capstone business policy/strategic management course sections that employed the Micromatic all-enterprise business simulation (Scott & Strickland, 1985). The practitioner data were obtained by mailing the same questionnaire to approximately 550 employed business alumni who, as college seniors (in the five years prior to the survey), were also exposed to the same simulation by the same instructors. Variables and Measurement Overall educational effectiveness of simulation gaming as an instructional pedagogy was measured through a 7-point scale item ranging in response options from 1 = not at all effective, to 7 = most effective. Effectiveness of simulation gaming in teaching the various managerial skills (i.e., teamwork/interpersonal, theory application to real world situations, planning, conceptual/integrative, problem solving/decision making, and environmental analysis) were operationalized through multi-item Likert scales. Specifically, for each of the above variables, respondents were first asked to rate their agreement/disagreement (on a 7-point scale) with a number of statements concerning effectiveness of the business simulation in developing their skills in that area. Responses on related items were then aggregated to compute the respondent’s score on the corresponding variable. Chronbach’s alpha reliability coefficients on these items were quite high and ranged from 0.71 for conceptual/integrative to 0.88 for problem solving/decision waking. Next, respondents were asked how important (1 = not at all important, 6 = extremely important) they thought issues presented in the same questionnaire items (as above) were/will be to their performance on the job. Again, related items were aggregated to compute scores on the importance of corresponding management skills. Cronbach’s alpha reliability coefficients for these variables were also high and ranged from 0.67 for conceptual/integrative to 0.86 for planning. Data Analysis Analysis of variance was the primary method of data analysis for this study. One-way ANOVA was applied to the data to test the differences in mean values of the variables for the student and practitioner groups. RESULTS AND DISCUSSION Results of the one-way ANOVA on simulation gaming’s overall effectiveness and effectiveness in teaching specific management skills are presented in Table 1. TABLE 1 Effectiveness of Simulation Gaming in Teaching Managerial Skills: Results of One-Way ANOVA Managerial Skills: Means F-value Students (n = 146) Practitioners (n = 124) Teamwork/ Interpersonal 4.65 5.22 14.91*** Theory Application 4.28 4.46 1.35 Planning 4.67 4.91 2.67* Conceptual! Integrative 4.44 4.67 1.92 Problem Solving/ Decision Making 4.43 4.72 3.68** Environmental Analysis 4.78 4.89 0.47 Overall Effectiveness 4.52 5.11 7.13*** * p < 0.10 ** p < 0.05 *** p < 0.01 In general, both student and practitioner subjects have reacted favorably to the pedagogical benefits of Development In Business Simulation & Experiential Exercises, Volume 18, 1991 55 the simulation experience. The mean ratings in all managerial skill areas are larger than 4 (on a 1 to 7 scale) for both groups. It also is clear from the table that the mean rating of overall simulation effectiveness is significantly higher (p = < 0.01) for practitioners than it is for students. Furthermore, mean scores for practitioners in all specific skill areas are consistently higher than those for students. In addition, the differences between mean scores for the two groups on teamwork/interpersonal, planning, and problem solving/decision making are statistically significant (p = < 0.01, p = < 0.1, and p - < 0.05, respectively). Finally, comparing the mean scores of various skills for the practitioner group indicates that this group views the opportunity to work in a team context and the interpersonal skills learned from it as the most tangible outcome of the simulation experience. All of this clearly suggests that practitioners, as compared to students: a) attribute significantly greater overall educational utility to business simulation gaming as a teaching pedagogy, and b) view business simulation as particularly more effective in teaching such essential managerial skills as teamwork, planning, and problem solving/decision making. In short, results of this study seem to provide strong support for the hypothesized relationships implied in Proposition 1 stated earlier. It appears that the practitioners’ maturity and work experience provide them with a significantly greater appreciation for the true value of collegiate simulation experiences. The above results also indicate that some of the inconsistencies in the findings of previous research with respect to simulation gaming effectiveness are, in fact, attributable to the differences in the type of study participants used by different researchers. Table 2 presents the results of ANOVA on importance of various management skills to job performance as perceived by students and practitioners. The results from Table 2 appear to partly support the relationships hypothesized in Proposition 2 stated above. According to this table, differences in student and practitioner mean scores on teamwork/interpersonal and environmental analysis are both statistically significant (p = < 0.01 and p = < 0.1, respectively). That is, when asked to rate the importance (to job performance) of managerial skills that were developed through the simulation, practitioners reported teamwork/interpersonal skills to be significantly more important than did students, while students perceived environmental analysis skills to be more important than did their practitioner counterparts. This result suggests that, compared to students, practitioners attach significantly greater importance to interpersonal/teamwork skills as a requirement for effective performance on the job. The fact that students (in comparison with practitioners, who are more realistic and speak from experience) underestimate the significance of such skills is a reflection of their naivete and bias stemming from lack of real- world professional experiences. Table 2 also indicates that students perceive environmental analysis to be significantly more important to job performance than do practitioners. This ‘nay be attributable to a combination of: a) the competitive nature of the simulation experience requiring participants’ constant awareness of competitive and other environmental factors, b) the recency of the students’ experience in the competitive simulation exercise, and c) predominance of entry- TABLE 2 Importance of Various Managerial Skills to Job Performance: Results of One-Way ANOVA Means Managerial Skills: Students Practitioners (n = 146) (n = 124) F-value Teamwork/ Interpersonal 4.61 4.93 7.59*** Theory Application 4.31 4.27 0.10 Planning 4.54 4.71 1.83 Conceptual/ Integrative 4.47 4.46 0.01 Problem Solving/ Decision Making 4.56 4.70 1.60 Environmental Analysis 4.54 4.31 2.84* *p < 0.10 **p < 0.05 ***p < 0.01 to mid -level managers and business professionals in our practitioner subsample (environmental/competitive forces may not be highly relevant to job responsibilities at these organizational levels). REFERENCES Biggs, W. D. & Greenlaw, P. S. (1976) The role of information in a functional business game. Simulation & Games, 7: 53-64. Butler, R. J., Pray, T. F. & Strang, D. R. (1979) An extension of Wolfe’s study of simulation game complexity. Decision Sciences, 10:480-486. Cangelosi, V. E. & Dill, W. R. (1965) Organizational learning observations toward a theory. Administrative Science Quarterly, 10: 175-203 Chisholm, T. A., Krishnakumar, P. & Clay, J. P. (1980) Are computer simulations sexist? In D. C. Brenenstuhl and W. D. Biggs (eds.) Proceedings of the Seventh Annual Conference of the Association for Business Simulation and Experiential Learning, 7:160-164 Faria, A. J. (1987) A survey of the use of business games in academia and business. Simulation and Games, 18: 207-224 Hemmasi, M., Graf, L. A. & Kellogg, C. E. (1989) A comparison of the performance, behaviors, and analysis strategies of MBA versus BBA students in a simulation environment. Simulation and Games, 20: 15-30. Development In Business Simulation & Experiential Exercises, Volume 18, 1991 56 Horn, R. E. & Cleaves, A. (1980). The Guide to Simulation/Games for Education and Training Newbury Park, CA: Sage. Keys, B. (1976). A review of learning research in business gaming. In B. H. Sord (ed.) Proceedings of the Third Annual Conference of the Association for Business Simulation and Experiential Learning Miles, W. G., Jr., Biggs, W. D. & Schubert, J. N. (1986) Student perceptions of skill acquisition through cases and a general management simulation: A comparison Simulation and Games, 17: 7-24. Parasuraman, A. (1981) Assessing the worth of business simulation games. Simulation and Games, 12: 189-200 Partridge, S. E. & Sculli, D. (1982) Management skills and business games. Simulation and Games, 13: 232-239 Roderick, R. D., Wilterding, J. A. & Eldredge, D (1979). Simulation as a technique for teaching collective bargaining. Proceedings of the Sixth Annual Conference of the Association for Business Simulation and Experiential Learning, 6: 56-59. Scott, C. R., Jr. & Strickland, A. J. III (1985) Micromatic: A Management Simulation. Boston: Houghton Mifflin Starbuck, W. H. & Kobrow, E. (1966) The effects of advisers on business game teams. American Behavioral Scientist, 10: 28- 30. Stone, E. F. (1982). Research design issues in studies assessing the effects of management education. Proceedings, National Academy of Management Wolfe, J. (1978) The effects of game complexity on the acquisition of business policy knowledge Decision Sciences, 9: 143-155 Wolfe, J. (1985) The teaching effectiveness of games in collegiate business courses: A 1973-1983 update. Simulation and Games, 16: 251-288 Wolfe, J. & Roberts, C. R. (1986) The external validity of a business management game: A five-year longitudinal study. Simulation & Games, 17: 45-59. Table of Contents Volume 18, 1991 Personality Types and Total Enterprise Simulation Performance Using DIS 'n DAT as a Decision Support System for a Marketing Simulation Game Theoretical Derivation of a Market Demand Function for Business Simulators The Ethnographic Case Study: An Experiential Approach to Teaching Retail Management Electronic Bulletin Board Systems (BBS): Support Software for Computer Simulations The New Budget Game Negame: A Cross-Cultural Role-Play to Introduce Students to the Familiarization Stage of Negotiations Modeling Short-Run Cost and Production Functions Using Sheppard's Lemma in Computerized Business Simulations Increasing Simulation Realism through the Modeling of Step Costs Predicting Simulation Performance: Differences Between Groups and Individuals A Facility Location Case to Stimulate Classroom Interaction Educational Effectiveness of Business Simulation Gaming: A Comparative Study of Students and Practitioner Perspective Ethical Dilemmas in Experiential Learning: Issues and Strategies A Critical Review and Assessment of ABSEL's Award-Winning Procedures and Protocols Political Risk: A Simulation for Business Practitioners Upside Down: A Cross-Cultural Game in Experiential Learning Gorby's Dilemma: From Communism to Free Enterprise in Two Hours Strategic Market: Planning with the COMPLETE Product Portfolio Analysis Package: A Marketing Decision Support System Career Concepts and Total Enterprise Simulation Performance Experiential Learning in Human Resources: A Performance Appraisal Application Managerial Motivation and Realism Among MBA Student as Viewed through The Looking Glass, Inc. Simulation An Experiential Approach to Teaching Data Analysis Using MYSTAT: Rationale, Procedures and Results Practicing What Was Preached: A Sequential Learning Model put to the Test Student Attitudes about Policy Course Simulations An Investigation of the Relationship Between Simulation Play, Performance Level and Recency of Play on Exam Scores The Effect of Leadership and Cognitive Processing Styles upon Peer Performance Evaluation: Implication for the Utilization of Simulations in Business Pedagogy On the Transfer of Market Oriented Business Games to Socialist Cultures An Application of Financial Analysis of the Business Firm in a Simulated Competitive Environment Collective Bargaining Simulation: An Exercise based on a Familiar Theme Making Business Policy a Strategic Management Experience A Student Exercise for Intergrating the Concepts of Power and Motivation The Boundaries Extended: An Experiment Comparing Dialectical Inquiry, Devil's Advocacy and Consensus Using the Executive Game Using a Simulation Package to Develop a Simulation Exercise in Cost Accounting Success Factors in Experiential Training for Creative Problem-Solving Teams An Experiential Approach for teaching Quality management Critical Success Ratios: A Comparison of Two Business Simulations in a Multi-Year Environment Stocklogs: A Classroom Exercise for Teaching the Logistical Relationship of Location and Inventory The Organizational Leadership Program Simulating Business Decision-Making: Using Statistical Cases for Classroom Exercises Scripting for the Classroom Upgrading the Business Strategy and Policy Game Developing Student Team-Building and Leadership Skills Using Computer-Aided Experiential learning Strategies Organizing and Outward Bound Field Trip An Architecture for Extensible Simulation Games Performance in the Capstone Business Course: What is the effect of Pedagogy, Learning Styles, and Student Motivation? Operational Strategy with Participant-Modifiable Parameters An Example of a Personal Selling Case Transformed into a Role Play Scenario Instructional Software: It's Evolution and Current State of the Art in the Business Curriculum Ascertaining Performance Variables for use in Determining Student's Grades in Courses Employing a Business Simulation Designing Management Seminars Using Business Simulations The Accounting Information Systems Course: Bridging the Gap between the Classroom and the Real World The Good Cooks Guide to Training Excellence: Working with Passion A Demonstration on Multiple Data Collection Methods: Seeing Strategic Issues Through the Looking Glass Simulation Systems Analysis and Design: Why Undergraduate Education gets a Failing Grade Accommodating Organizational Culture: An Evaluation of Management Development Delivery Modes in Varying Organizational Cultures Modeling Total Quality into Business Simulations The Political Futures Game Meeting Meeting Objectives