AN INVESTIGATION OF THE RELATIONSHIP BETWEEN SIMULATION PLAY, PERFORMANCE LEVEL AND RECENCY OF PLAY ON EXAM SCORES Development In Business Simulation & Experiential Exercises, Volume 18, 1991 111 AN INVESTIGATION OF THE RELATIONSHIP BETWEEN SIMULATION PLAY, PERFORMANCE LEVEL AND RECENCY OF PLAY ON EXAM SCORES William J. Wellington, University of Windsor A. J. Faria, University of Windsor ABSTRACT The present study examined the relationship between simulation participation, level of performance in a simulation competition, and recency of play with exam scores in a principles of marketing course. The rigorously controlled experiment involving 389 students found no relationship between simulation play and exam scores, level of simulation performance and exam scores, and recency of simulation play and exam scores when CPA was controlled for. INTRODUCTION There are a number of instructional methods that can be used to teach business courses. Lectures and reading assignments tend to serve as the foundation of most, if not all, such courses. As a supplement, cases or simulation games are often added to the course. When adding supplemental material to the course, however, it is essential for the instructor to address two major issues: the pedagogical value of the technique and the time commitment that will be required by both students and instructor. Despite the proliferation and widespread use of business simulation games, a review of the literature reveals that the pedagogical value of such games still remains unclear. Are such games a needless time-consuming activity for students and instructors, or are they an effective vehicle for the achievement of specific educational objectives? The present study sought to determine if incorporating a business simulation game in a principles of marketing course improves the acquisition of marketing knowledge. Other potential benefits of game playing, such as improving interpersonal skills or developing analytical skills were not investigated. PAST RESEARCH Accompanying the development of business games and their increased use has been an active research track. Such research has generally been concerned with identifying factors affecting the simulation learning environment and simulation performance, the learning aspects of simulation games, and the relative educational benefits of simulation games versus other approaches to teaching. Within this last area, four major review articles have appeared (Greenlaw and Wyman 1973; Keys 1976; Wolfe 1985; and Miles, Biggs and Shubert 1986). Greenlaw and Wyman (1973) reviewed 22 articles published from 1961 to 1972 and concluded that little evidence existed as to whether simulation games were a superior, or even adequate, method of instruction. After reviewing 39 studies published from 1973 to 1983, Wolfe (1985) also determined that definitive conclusions about gaming effectiveness could not be reached. Miles, Biggs and Shubert (1986) concluded (based on 16 studies reviewed) that, while students believe they learn as much or more from business games as from cases, the results are mixed and difficult to interpret because of the wide variety of study environments employed. More in line with the research to be reported here, Keys (1976) reviewed fifteen studies that compared simulation game sections of a class with sections using a non-simulation form of instruction. The review was limited to studies that used some form of end-of- course examination as the focus of the comparison. Six of the comparisons involved end-of--course case analyses; six involved essay final exams; two used a true-false final exam; and one a multiple-choice final exam. In the six studies using a case final exam, the reported results showed no performance difference between the simulation and nonsimulation sections in four of the studies while the simulation section outperformed the non-simulation section in the other two. Where an essay final exam was used, the simulation section students scored higher in four instances, the case students higher in one, and there was no performance difference in the other. The simulation students scored higher in both studies using true-false final exams while the case students scored higher in the one study using a multiple-choice final exam. The results from the 92 studies reported on by Greenlaw and Wyman (1973), Keys (1976), Wolfe (1985) and Miles, Biggs and Shubert (1986) would suggest, at a minimum, that simulation games are as useful as other approaches to teaching business courses. These studies do not, however, provide any evidence as to whether adding a simulation game to existing course material enhances student learning of that material. Shedding some light on this issue, though, are three recent studies (Whiteley and Faria 1989; Anderson and Lawton 1990; and Faria and Whiteley 1990). Whiteley and Faria (1989) reported results from a study involving 190 students In a marketing class. Half of the students played a simulation game (and were responsible for all other class assignments) while the other half did not. A multiple-choice final exam was used in the class. The overall final exam scores of the simulation players were slightly higher, although not significantly different, than those of the nonsimulation players. When examining question types on the final exam, Whiteley and Farla (1989) reported that there was no difference in scores on theoretical and applied type questions but the simulation students scored significantly higher on quantitative questions. A second study by Faria and Whiteley (1990) used the same study design except that the students in the marketing course were divided into three groups. One group played the simulation game in teams of three members, the second group played as Development In Business Simulation & Experiential Exercises, Volume 18, 1991 112 individual person teams, while the third group did not play the simulation. Again, no significant difference in overall final exam score was found (although the simulation players scored slightly higher than the nonplayers). By question type, simulation team players scored significantly higher than the nonplayers on the quantitative questions while individual simulation players scored significantly higher than nonplayers on the applied quest tons. Anderson and Lawton (1990), using 40 students in a business policy course, examined the relationship between performance in a simulation game and final exam scores. All students participated In the simulation competition and the final exam was of an essay nature. No significant relationships were found between performance in the simulation (as measured by net income, ROI and ROA) and final exam scores. PURPOSE AND HYPOTHESES The findings from previous research are inconclusive as to the learning effectiveness of business simulation games. The purpose of the present study is to attempt to clarify, from a learning point- of-view, the appropriateness of using this instructional approach in a principles of marketing course. The present study will examine three Issues: (1) simulation play versus no play and the relationship, if any, to course exam scores; (2) the relationship between simulation performance level and exam scores; and (3) the relationship between recency of simulation play and exam scores. Whiteley and Faria (1989) and Faria and Whiteley (1990) reported no significant relationship between simulation play and overall multiple choice final exam scores. Anderson and Lawton (1990) reported no significant relationship between level of performance in a simulation game and final exam scores. No previous research has reported on recency of simulation play and exam scores. Thus, the following hypotheses have been formulated for testing purposes. Hi: There will be no significant relationship between participation in a simulation competition and exam scores H2: There will be no significant relationship between level of performance in a simulation competition and exam scores. H3: There will be no significant relationship between recency of participation in a simulation competition and exam scores. The first two hypotheses are based on findings from the limited previous research while the third naturally follows from the first two. METHODOLOGY The subjects for the research to be reported here were 389 students in two sections of a principles of marketing class. Both sections were taught by the same instructor, used the same textbook, viewed the same videos, and took common multiple choice midterm and final exams. The simulation game used was LAPTOP: A Marketing Simulation (Faria and Dickinson 1987). Students were divided into teams of three or four players. In total, 107 simulation teams were formed and divided into 18 industries of approximately six teams each. In one section of the class, the simulation competition was started at the beginning of the semester and completed just prior to the midterm exam. In the other section, the simulation was started just after the midterm and completed just prior to the final exam. In each section, six sections were made in the simulation competition and the simulation grade counted towards 15 percent of the final grade in the course. In addition to making decisions in the competition, the students were required to complete a self- report attitude survey with each simulation decision. Among other things, the survey measured time spent making each simulation decision; expected team ranking at the end of the competition; team cohesiveness; simulation enjoyment; simulation experience relative to lectures, cases, and readings; perceived appropriateness of the simulation evaluation method being used; the degree to which the students felt that the simulation performance reflected their managerial abilities; perception of the benefits of group work; a rating of each group member’s contribution to the simulation; and team organization. Each students overall grade point average was also obtained from university records. Finally, simulation performance was measured in four ways: final team ranking within the industry, cumulative earnings per share, a relative earnings measure termed the EPS gap, and a computed simulation performance grade based on the previous three measures. A t-test was performed comparing midterm examination performance, final examination performance, grade point averages, final LAPTOP performance, and attitudes towards the simulation competition. This test was designed to discover differences between the two simulation groups. FINDINGS Attitudes concerning team cohesiveness (4 Item scale), simulation enjoyment (3 item scale), and simulation games compared to other instructional. methods (3 item scale comparing the simulation to cases, lectures, and readings) were measured for each of the six decision periods using multi-item scales. The internal consistency reliability of the scales was measured using co-efficient alpha. All of the scales had average values in excess of .80 (see Table 1) which is acceptable for basic research (Nunally 1978, p.245). TABLE 1 ALPHA RELIABILITY OF MULTI-ITEM SCALES Scale Name Mean Reliability Range Team Cohesiveness .9271 .9040 - .9471 Simulation Enjoyment .8110 .7470 - .8638 Simulation Compared to Methods Other Instructional .8692 .7929 - .9330 The findings of the t-test (see Table 2) comparison of the two groups (premidterm and postmidterm players) revealed that they did nor differ significantly on any of the attitude measures at the end of the simulation competition. The only significant differences occurred with respect to midterm exam performance, final exam performance and GPAs. Development In Business Simulation & Experiential Exercises, Volume 18, 1991 The findings that the midterm and final exam scores differed between the premidterm and postmidterm groups was encouraging in that it suggested that simulation play and examination performance might be related. Furthermore, a correlational analysts (see Table 3) between midterm and final exam scores versus the simulation performance measures indicated a significant but weak relationship between simulation performance and exam performance for the premidterm players. However, the correlational analysis indicated that there was no relationship between simulation performance and exam performance for the postmidterm players. premidterm and postmidterm groups were significantly different and thus support the t-test results. With respect to the rank order performance by class section, the premidterm group exhibited midterm and final exam performance differences according to the one-way analysis of variance results. The findings for the postmidterm simulation players were that midterm exam performance was significantly different for the rank order groups but there was no significant difference in the final exam performance. TABLE 2 T-TEST VALUES FOR PREMIDTERM AND POSTMIDTERM PLAYERS Variable Pretmid Mean Postmid Mean t-value Significan ce Midterm Exam Score 56.45 54.61 2.14 .034 Final Exam Score 67.38 65.19 2.24 .026* Grade Point Average 70.64 68.05 2.58 .010k LAPTOP Grade 6.49 6.41 .44 .66 1 End EPS -6.81 -3.58 -.63 .529 End Ranking 3.34 3.51 -.98 .330 Expected Ending Rank 2.33 2.44 -1.23 .219 End EPS Gap 33.37 37.03 -.75 .453 End Cohesiveness 1 .89 2.01 -.97 .321 End Enjoyment 2.43 2.56 -.84 .401 End Compared to Cases, Readings and Lectures 2.84 2.87 -.16 .869 End Evaluation Method 3.07 3.00 .40 .693 End Managerial Ability 3.20 3.50 -1.60 .111 End Group-work Attitude 4.73 4.45 1.30 .196 Average Decision Time 96.07 94.75 .28 .781 TABLE 3 CORRELATIONS BETWEEN CPA, EXAMINATION PERFORMANCE AND SIMULATION PERFORMANCE Variable Type Mid GPA Lap EPS ARank GAP T-Final .6856* .6140* .1506* .1115* -.1644* -.0736 S1-Final .6699* .6029* .2338* .1730* -.2245* -.0844 S2-Final .7001* .6166* .0089 .0430 -.0489 -.0520 T-Midterm .6352* .0845 .0572 -.1039 -.0350 S1-Midterm .6731* .1460* .1373* -.1560* -.0720 S2-Midterm .5605* -.0192 -.0343 -.0040 .0200 T-GPA .1415* .0375 -.1539* -.0223 S1-GPA .1500* .0830 -.1640* -.0089 S2-GPA .1220 -.0123 -.1206 -.0290 T-Lapgrade .5571* -.9279* .5955* S1- Lapgrade .5680* -.9365* -.6208* S2-Lapgrade .5539* -.9140* -.5731* T-EPS -5305* -.8883* S1-EPS -.5299* -.6230* S2-EPS -.5498* -.8230* T-Actual Rank .5487* S1-Actual Rank .5543* S2-Actual Rank .5527* * Significant at < .05 level. While exam performance differed between premidterm and postmidterm simulation players, this difference could be accounted for by differences in CPA. Therefore, a MANOVA analysts was undertaken to control for the effect of CPA on exam performance relative to the simulation treatments and for rank simulation performance relative to exam performance. Rank order performance was used because it was a categorical variable and correlated very highly with the the final LAPTOP grade. The initial one-way analysis of variance results (see Table 4) comparing group membership to midterm A MANOVA procedure was employed to control for the effect of GPA which was found to be significantly different between the premidterm and postmidterm groups. In all cases, the findings (see Table 4) indicate that group differences evident in the one-way analysis of variance procedures could be explained when the effect of CPA was controlled ____ for. This meant that premidterm or postmidterm play of a simulation game did not have a direct impact on exam performance. Furthermore, exam performance and simulation performance (as measured by rank order) were not related. DISCUSSION AND CONCLUSIONS The initial t-test results showed that those students playing the simulation game prior to the midterm exam scored higher on the midterm than those students who had not yet started the simulation competition; students who played the simulation game prior to the midterm also scored higher on the final exam; and those students who were on better performing teams in the simulation competition scored higher on the midterm and final exams. These findings might suggest some relationship between simulation participation and level of performance and performance on multiple GAP choice exams in a principles of marketing course. However, when CPA was controlled for, the results from the MANOVA analysis showed that there was no significant differences in midterm exam scores between the premidterm and postmidterm groups, no difference in final exam scores, and no relationship between exam performance level and simulation performance level. Thus) similar to the findings from past studies, all three hypotheses are accepted. The acceptance of these three hypotheses supports the notion that simulation play and multiple choice exam performance are not related. These findings suggest that the kind of learning measured by multiple choice exams differs from the kind of learning that occurs from the play of simulation games. This is not surprising since the multiple choice exams used in most introductory level courses are primarily designed to measure recognition of basic concepts and principles while simulation play is designed to develop decision making skills. This suggests that simulation play involves skills which may not be directly measurable by normal multiple choice exams. With respect to the marketing class used for this study, simulation play was designed to measure the application of marketing principles and the quality of decision making and not the ability to remember basic principles. However, there is an alternative explanation for the results, which involves the units of analysts for the research. Simulation play is a group process while Development In Business Simulation & Experiential Exercises, Volume 18, 1991 113 TABLE 4 MANOVA RESULTS CONTROLLING FOR CPA F Sig. Final Exam Score by Treatment Group: Treatment Group Versus Final Exam Score One-way ANOVA Results 5.28 .022* MANOVA Results With GPA as a Covariate Regression with Covariate 224.01 .000* Treatment Group .76 .385 Interactions Between Covariate and Treatment Group SPA 210.64 .000* Treatment Group .14 .713 SPA by Treatment Group .06 .800 Midterm Exam Score By Treatment Group: Treatment Group Versus Midterm Exam Score One-way ANOVA Results 4.94 .027* MANOVA Results With GPA as a Covariate Regression with Covariate 253.77 .000* Treatment Group .53 .466 Interactions Between Covariate and Treatment Group GPA 227.73 .000* Treatment Group .48 .490 GPA by Treatment Group .63 .426 Midterm Exam Score by Rank and Section: Rank in Section 1 Versus Midterm Exam Score One-way ANOVA Results 2.20 •055* MANOVA Results With SPA as a Covariate Regression with Covariate 200.24 .000* Rank 1.50 .190 Interactions Between Covariate and Treatment Group SPA 191.00 .000* Rank .32 .900 GPA by Rank .31 .905 Midterm Exam Score By Rank and Section: Rank in Section 2 Versus Midterm Exam Score One-way ANOVA Results 2.85 .0l8* MANOVA Results With GPA as a Covariate Regression with Covariate 51.39 .000* Rank 1.86 .105 Interactions Between Covariate and Treatment Group GPA 46.64 .000* Rank .40 .848 SPA by Rank .54 .748 Final Exam Score By Rank and Section: Rank in Section 1 Versus Final Exam Score One-way ANOVA Results 3.28 .007* MANOVA Results With SPA as a Covariate Regression with Covariate 130.87 .000 Rank 1.96 .084 Interactions Between Covariate and Treatment Group SPA 127.63 .000k Rank .58 .719 SPA by Rank .50 .779 Final Exam Score By Rank and Section: Rank in Section 2 Versus Final Exam Score One-way ANOVA Results 1.08 .374 MANOVA Results With SPA as a Covariate Regression with Covariate 130.87 .000k Rank 1.96 .084 Interactions Between Covariate and Treatment Group SPA 127.63 .000k Rank .58 .719 GPA by Rank .50 .779 Probability C .05 Development In Business Simulation & Experiential Exercises, Volume 18, 1991 114 examination performance is an individual process. In this vein, though, Faria and Whiteley (1990) have looked at individual simulation performance versus multiple choice exam performance and report no significant relationship. The implications of the findings in this study suggest that the issue of the pedagogical value of simulations should revolve around how simulation play affects the development and acquisition of decision-making skills and interpersonal communication skills as opposed to the acquisition of business principles and knowledge. REFERENCES Anderson, P. H. and L. Lawton (1990), “The Relationship Between Financial Performance and Other Measures of Learning on a Simulation Exercise”, Proceedings of the Seventeenth Annual Conference of the Association for Business Simulation and Experiential Learning, 17, 6-10 Faria, A. J. and J. R. Dickinson (1987), LAPTOP: A Marketing Simulation, Homewood, Illinois: Richard D. Irwin, Inc Faria, A. J. and T. R. Whiteley (1990), “An Empirical Evaluation of the Pedagogical Value of Playing a Simulation Came in a Principles of Marketing Course”, Proceedings of the Seventeenth Annual Conference of the Association for Business Simulation and Experiential Learning, 17, 53-57. Greenlaw, P. S and F. P. Wyman (1973), “The Teaching Effectiveness of Games in Collegiate Business Courses”, Simulation & Games, 4 (2), 2 59-294. Keys, B. (1976), “A Review of Learning Research in Business Gaming”, Proceedings of the Third Annual Conference of the Association for Business Simulation and Experiential Learning, 3, 25 - 32 Miles, W. C., W. D. Biggs and J. N. Schubert (1986), “Student Perceptions of Skill Acquisition Through Cases and a General Management Simulation”, Simulation & Games, 10 (1), 75 - 86. Nunally, J. C. (1978), Psychometric Theory, New York: McGraw- Hill Book Company, Inc. Whiteley, T. R. and A. J. Faria (1989), “A Study of the Relationship Between Student Final Exam Performance and Simulation Came Participation”, Simulation & Games, 20 (1), 44-64. Wolfe, J. (1985), “The Teaching Effectiveness of Games in Collegiate Business Courses: A 1973–1983 Update”, Simulation & Games, 16 (2), 251-288. Table of Contents Volume 18, 1991 Personality Types and Total Enterprise Simulation Performance Using DIS 'n DAT as a Decision Support System for a Marketing Simulation Game Theoretical Derivation of a Market Demand Function for Business Simulators The Ethnographic Case Study: An Experiential Approach to Teaching Retail Management Electronic Bulletin Board Systems (BBS): Support Software for Computer Simulations The New Budget Game Negame: A Cross-Cultural Role-Play to Introduce Students to the Familiarization Stage of Negotiations Modeling Short-Run Cost and Production Functions Using Sheppard's Lemma in Computerized Business Simulations Increasing Simulation Realism through the Modeling of Step Costs Predicting Simulation Performance: Differences Between Groups and Individuals A Facility Location Case to Stimulate Classroom Interaction Educational Effectiveness of Business Simulation Gaming: A Comparative Study of Students and Practitioner Perspective Ethical Dilemmas in Experiential Learning: Issues and Strategies A Critical Review and Assessment of ABSEL's Award-Winning Procedures and Protocols Political Risk: A Simulation for Business Practitioners Upside Down: A Cross-Cultural Game in Experiential Learning Gorby's Dilemma: From Communism to Free Enterprise in Two Hours Strategic Market: Planning with the COMPLETE Product Portfolio Analysis Package: A Marketing Decision Support System Career Concepts and Total Enterprise Simulation Performance Experiential Learning in Human Resources: A Performance Appraisal Application Managerial Motivation and Realism Among MBA Student as Viewed through The Looking Glass, Inc. Simulation An Experiential Approach to Teaching Data Analysis Using MYSTAT: Rationale, Procedures and Results Practicing What Was Preached: A Sequential Learning Model put to the Test Student Attitudes about Policy Course Simulations An Investigation of the Relationship Between Simulation Play, Performance Level and Recency of Play on Exam Scores The Effect of Leadership and Cognitive Processing Styles upon Peer Performance Evaluation: Implication for the Utilization of Simulations in Business Pedagogy On the Transfer of Market Oriented Business Games to Socialist Cultures An Application of Financial Analysis of the Business Firm in a Simulated Competitive Environment Collective Bargaining Simulation: An Exercise based on a Familiar Theme Making Business Policy a Strategic Management Experience A Student Exercise for Intergrating the Concepts of Power and Motivation The Boundaries Extended: An Experiment Comparing Dialectical Inquiry, Devil's Advocacy and Consensus Using the Executive Game Using a Simulation Package to Develop a Simulation Exercise in Cost Accounting Success Factors in Experiential Training for Creative Problem-Solving Teams An Experiential Approach for teaching Quality management Critical Success Ratios: A Comparison of Two Business Simulations in a Multi-Year Environment Stocklogs: A Classroom Exercise for Teaching the Logistical Relationship of Location and Inventory The Organizational Leadership Program Simulating Business Decision-Making: Using Statistical Cases for Classroom Exercises Scripting for the Classroom Upgrading the Business Strategy and Policy Game Developing Student Team-Building and Leadership Skills Using Computer-Aided Experiential learning Strategies Organizing and Outward Bound Field Trip An Architecture for Extensible Simulation Games Performance in the Capstone Business Course: What is the effect of Pedagogy, Learning Styles, and Student Motivation? Operational Strategy with Participant-Modifiable Parameters An Example of a Personal Selling Case Transformed into a Role Play Scenario Instructional Software: It's Evolution and Current State of the Art in the Business Curriculum Ascertaining Performance Variables for use in Determining Student's Grades in Courses Employing a Business Simulation Designing Management Seminars Using Business Simulations The Accounting Information Systems Course: Bridging the Gap between the Classroom and the Real World The Good Cooks Guide to Training Excellence: Working with Passion A Demonstration on Multiple Data Collection Methods: Seeing Strategic Issues Through the Looking Glass Simulation Systems Analysis and Design: Why Undergraduate Education gets a Failing Grade Accommodating Organizational Culture: An Evaluation of Management Development Delivery Modes in Varying Organizational Cultures Modeling Total Quality into Business Simulations The Political Futures Game Meeting Meeting Objectives