Design and Effectiveness of a Self-Study Pedagogical Approach to Using a Simulation Game in the Classroom Page 136 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 ABSTRACT An innovative pedagogy is presented in which a simulation is designed and used as a self-study where students “learn by do- ing” on their own and at their own pace without direct instruc- tor involvement. The learning effectiveness of the simulation self-study approach is tested in an economics course with re- spect to how it impacts student performance. It was found that the self-study, as a supplemental tool, had a significant and positive impact on student exams and the course grades. INTRODUCTION There has been an ongoing debate, since the early 1970s on the pedagogical effectiveness of simulations. A study by Gold (2015) presented a case arguing that the stream of research on the pedagogical effectiveness of simulations should change fo- cus. The relevant question that needs to be addressed is not whether simulations are effective learning tools; enough studies have supported this hypothesis as evidenced by a comprehen- sive literature review on the subject by Wellington, Hutchinson, and Faria (2010). What we need to explore in further detail is the question: What are the most effective ways to use or inte- grate a simulation in a classroom? As highlighted by Goosen (2002), the degree of learning is highly dependent on the way in which a simulation is administered and integrated into the class by the instructor. Although research has been done on simula- tion pedagogy, advancements in computer technology with re- spect to accessibility, speed, and cost-reductions have opened up new possibilities for creative alternative teaching methods with respect to simulation use and, as a consequence, further inquiry in this area is warranted. The primary purpose of this paper is to introduce an inno- vative “self-study” approach for integrating a simulation into a classroom that takes advantage of some of the latest technolo- gies and then testing the learning effectiveness of this approach. The paper proceeds by: (a) explaining the design of the simula- tion self-study method used in this study; (b) testing the learn- ing effectiveness of the simulation self-study; and (c) assessing the results of the tests. DESIGN OF THE SIMULATION SELF STUDY The basic idea of the simulation self-study is to allow stu- dents to play simulation games on their own with structured assignments that integrate exercises with the simulation that link to the theories being taught in the class. The students com- plete the games and exercises on their own and at their own pace, but subject to the time constraints established by the in- structor. The number of assigned games and exercises would depend on the courses objectives established by the instructor. The simulation self-study pedagogy explored in this paper is designed and developed for a course in economics at either the foundation or advanced (intermediate) levels using a web- based game called “Beat the Market Online: A Microeconomic Game” which may be accessed at http:// www.goldsimulations.com. The major difference in the self- study between the two levels is the complexity of the simulation games used with respect to such factors as: the number of firm decisions, the number of firms competing in the marketplace, whether the economy is stable or changing, and if there are ran- dom events occurring. Leaderboard To motivate students to do well in the self-study simulation games and recognize their performance, there is an online lead- erboard that broadcasts their accomplishments. Entry onto the online Leaderboard is reserved for the TOP 10% of all play- ers, both nationally and internationally, not only in one course. The real-time leaderboard activity may be viewed by both play- ers and non-players at http://www.goldsimulations.com/ leaderboard. Players may elect to use anonymous or alias names. Self Study Game Story and Learning Progression Students begin their self-study journey by managing a mo- nopoly firm with no direct r ivals in a stable economy. The firm is producing a perishable product and is alerted to be care- ful not to overproduce. Students may repeat games until they succeed, and once they do the challenge increases! The monop- oly market will change to an oligopoly, and the number of rival firms in the market will increase to 9. Although there are some rivals in this market, each firm will have a significant market share and the ability to differentiate their product and build brand loyalty. But players are warned not to drop their guard because their rivals are also large and powerful, and if aggres- sive can have a substantial impact on their sales, which was not the case in the monopoly market. After players have proven their ability to succeed in the oligopoly, the challenge (and fun) continues to increase! Now they have the privilege of showing their worth (economic skills) by managing a firm in a highly competitive market (called Monopolistic Competition) starting with 25 rival firms! And to further add to the challenge, new firms may also enter this market and compete. In this case, the players will need to figure out how to keep the customers they have or attract more customers when the numbers of rival firms increase. In this environment the players are cautioned that they will need to carefully determine the right price to succeed ow- ing to the rising number of competitors. Once they have suc- ceeded in this market the simulation is over; but students may elect to replay (because learning is the key purpose). Design and Effectiveness of a Self-Study Pedagogical approach to using a Simulation Game in the Classroom Steven Gold Rochester Institute of Technology stevengold@saunders.rit.edu http://www.goldsimulations.com http://www.goldsimulations.com http://www.goldsimulations.com/leaderboard http://www.goldsimulations.com/leaderboard mailto:stevengold@saunders.rit.edu mailto:stevengold@saunders.rit.edu mailto:stevengold@saunders.rit.edu mailto:stevengold@saunders.rit.edu mailto:stevengold@saunders.rit.edu mailto:stevengold@saunders.rit.edu mailto:stevengold@saunders.rit.edu Page 137 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 Learning Activities Learning is further enhanced by uniquely combining simu- lation games with topic-specific simulation exercises. The topic- specific simulation exercises are very different than simulation games. In simulation games students manage a firm applying economic concepts to maximize profits whereas in the topic- specific simulation exercises they learn by observing what is happening in a set of pre-set example simulation games and then are asked a series of multiple choice questions or problems to test and reinforce their understanding of specific topics such as demand, equilibrium, price elasticity, production, costs and more. Both the simulation games and simulation exercises are automatically graded and the solutions provided to the students. To permit continued learning, exercises may be repeated, but the answers to the new exercises will be different because the parameter values used in the games will change (like price elas- ticities, average costs, and business cycles) with each new game. Repetition is considered to be important to help students learn from their mistakes. I. Introductory Level Games and Exercises A list of the recommended order for the students to com- plete the games and exercises in the self-study at the introducto- ry level is given in Table I. Referring to Table I, we suggest (order 1) students first do an Introduction Exercise which re- quires viewing the “Guided Tour on How to Play the Game”, and answering multi-choice questions about a pre-designed game. After this (order 2), students play Game 1 for practice, and then do two more exercises (order 3 and 4), including the short-run Monopoly and short-run Production Exercises. These exercises require students to play a game and answer multiple choice questions about the results in the game, focusing on the monopoly market and production relationships. Once students have done this, they are asked to repeat Game 1 (order 5) and see how much better they perform with their newly acquired knowledge. Games & Exercises may be repeated as many times as needed until students are satisfied with their understanding and performance. Remember that each new game repeated has different parameters (elasticity and productivity coefficients, etc.), so the results and answers to multiple choice questions would change, making repetition a valid learning experience. The self-study continues in a similar fashion by completing the remaining games and exercises in the order specified (6 to 17). The “leaderboard” mentioned previously is available for stu- dents to see how their performance ranks against other students. TABLE 1 RECOMMENDED ORDER FOR COMPLETING GAMES AND EXERCISES AT INTRODUCTORY LEVEL Order Simulation Games* Simulation Exercises 1 Getting Started: Introduction 2 Game 1 3 Short Run Monopoly 4 Short Run Production 5 Game 1 - Repeat 6 Price Elasticity 7 Law of Demand 8 Game 2 9 Market Equilibrium 10 Game 3 11 Game 4 12 Short Run Oligopoly 13 Game 5 14 Game 6 15 Short Run Monopolistic Competition 16 Game 7 17 Game 8 LEADERBOARD** * Number of rival firms increase as you advance from games 1 to 8 **Leaderboard results for all game are posted on the Gold Simulations website. Page 138 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 II. Advanced Level Games and Exercises The more advanced self-study procedure is shown in Table II and, similar to the introductory level (Table I) specifies the recommended order for completing games and exercises at this higher level. The differentiating feature at the higher level is the focus on the long-run, allowing firms to build capacity over time in the games and deal with changing economic conditions like business cycles. Creating Personal Games to Directly Challenge Other Students To add to the excitement of the self-study, students are giv- en the option to set-up their own “personal games” and ask oth- er students to compete against them. They may invite one or more (any number) of students to compete in a multi-player game environment where each student manages a firm in the same marketplace. These multi-player games are interactive and one student’s decisions for their firm (like changing price) will impact the sales of other students. Students may set up as many of these personal games that they want and can even select the level of complexity of the game by controlling the number of firms in the market and even the number of decisions controlled by the firms. ASSESSMENT OF LEARNING EFFECTIVENESS To test the effectiveness of the self-study in terms of stu- dent learning, a pilot study of one graduate level foundation economics course was evaluated. Each student was required to complete all the advanced level II games with the accompany- ing exercises. These are single player games where each student competes against computer-managed firms. The students were given the assignment at the beginning of the semester and were required to complete it by the end of the semester. This gave them the opportunity to do the self-study at their own pace; but was given a recommended schedule of times to do the games and exercises during the course. The self-study games and exer- cises were worth 15% of the course grade. The student’s perfor- mance on the self-study was then compared to their overall course grade which included all the other assignments and ex- ams in the class that tested the knowledge of the textbook and lecture material. The results are shown in Table III, which lists each student’s performance on the self-study ranked in order from the top rating to the bottom; along with the ranking of their exam grades and their overall course grades which includ- ed all assignments (except for the self-study). The exams tested theoretical material from the textbook, not the simulation. Re- ferring to Table III, casual observation indicates that students ranked on top in the self-study rating also did well in the class ranking on exam and overall course performance (but a statisti- cal test to assess this will follow). For example, Student “A” was ranked number 1 in the class in the self-study performance and number 2 in the exams and overall grade in the course. The worst performing student in the self-study (student T), ranked in last place (20th) on this measure and in the overall course grade; but did better on the exams, placing 13th. To measure the statistical significance of the relationship between the ranking of the self-study performance and the stu- dent’s overall course grade, a Spearman Rank Order correlation analysis was performed. The Spearman Rank Order correlation results are given in Table IV and show a statistically significant relationship between the performance ranking in the self-study and the exams, and also between the self-study and the overall course grade ranking. The magnitude of the Spearman rho cor- relations are between 0.47 and 0.58 with statistically significant confident levels exceeding 95% (P-values < 0.05). The statistical significance in the Spearman rank order cor- TABLE 2 RECOMMENDED ORDER FOR COMPLETING GAMES AND EXERCISES AT ADVANCED LEVEL Order Simulation Games* Simulation Exercises 1 Game 9 2 Long Run Production 3 Long Run Cost 4 Long Run Monopoly 5 Game 9 Repeated 6 Revenue Maximization Game 10 7 Game 11 8 Game 12 9 Long Run Monopolistic Competition 10 Game 13 11 Game 14 LEADERBOARD** * Number of rival firms increase as you advance from games 9 to 14 **Leaderboard results for all games are posted on the Gold Simulations website Page 139 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 relation analysis is consistent with the hypothesis that the self- study assignment helps the student learn the course material and may be used as a predictor of a student’s “overall” performance in the class. Yet, one may argue that the results show only cor- relation and not causation. Even if this is true, the results of the self-study has the additional benefit of being an early predictor of a student’s performance in the class, and may be used by the instructors to flag students that may need additional help early in the semester. As a further test of the impact of the self-study assignment, a linear regression was done to confirm the predictive ability of the student’s performance in the self-study with the overall course grade (net of the self-study assignment). The results are shown in Table V. The regression is statistically significant at the 1% level with an F-statistic of 9.58. The variation in the self-study grade explains 34.74% of the variation in the student’s overall course grade. This is considered to be an important finding since the self-study is only 15% of the student’s grade in the course. Al- so, the self-study rating coefficient of 0.669 is statistically sig- nificant at the 1% level based on a t-test statistic of 3.095. The estimated coefficient predicts that a 1% increase in the self- study performance rating will increase the students overall course grade (on assignments other than the self-study) by 0.67% which is a meaningful impact. This analysis also sup- ports the hypothesis that a self-study methodology for integrat- ing a simulation into the classroom has merit and appears to be an effective pedagogical approach. But this finding is based on limited data, in a pilot study in one course with only 20 stu- dents, and clearly further research on this methodology is need- ed. SUMMARY AND CONCLUSIONS This study resumes a stream of research examining the im- pact on student learning of using an educational simulation game in the classroom. The research question in this study is not whether learning occurs with a simulation, enough studies support this conclusion, but rather the pedagogical effectiveness of a specific approach. It has been argued that the way in which a simulation is used in the classroom is a significant determi- nant of its value to student learning. An innovative methodology has been presented where a simulation is designed and used as a self-study in which stu- dents “learn by doing” on their own, at their own pace, without direct instructor involvement. The students are required to complete a set of pre-designed simulation games and exercises on their own; and there success is highlighted on a leaderboard to enhance motivation. Theoretically, this draws upon the most powerful aspects of experiential learning. This approach to us- ing a simulation in the classroom also has the advantage of free- ing up instructor time (since students work on the simulation activity on their own), which gives the instructor more time for supportive teaching activities. As a pilot program, the self-study approach is tested in an TABLE 3 COMPARISON OF STUDENT SELF-STUDY AND EXAMS: RANKED BY SELF-STUDY PERFORMANCE Student ID Self-Study Exams Course* Rating Cl ass Rank Grade Cl ass Rank Overall Grade Class Rank A 99.74% 1 93.00% 2 95.19% 2 B 99.05% 2 90.00% 4 93.33% 3 C 98.54% 3 86.00% 9 89.18% 9 D 98.49% 4 96.00% 1 96.13% 1 E 97.69% 5 90.00% 5 84.07% 14 F 97.68% 6 85.00% 11 90.69% 5 G 95.64% 7 80.00% 16 85.03% 13 H 95.32% 8 87.00% 8 90.14% 7 I 94.95% 9 91.00% 3 89.32% 8 J 94.82% 10 81.00% 15 87.25% 10 K 94.53% 11 74.00% 19 82.07% 17 L 94.18% 12 82.00% 14 87.07% 12 M 93.03% 13 84.00% 12 90.18% 6 N 92.68% 14 74.00% 20 80.81% 18 O 87.70% 15 77.00% 17 83.55% 15 P 87.04% 16 76.00% 18 83.08% 16 Q 86.37% 17 88.00% 7 78.02% 19 R 86.19% 18 85.00% 10 87.10% 11 S 85.09% 19 89.00% 6 91.76% 4 T 83.82% 20 83.00% 13 73.28% 20 Page 140 - Developments in Business Simulation and Experiential Learning, Volume 43, 2016 economics course with respect to how it impacts student perfor- mance (grades) as a measure of learning effectiveness. The re- sults support that: (a) there is a significant relationship between student performance on the simulation self-study and their grades on exams that cover the textbook material. This is consistent with the hypothesis that the simulation self-study is helping students grasp the content specific theories of the course; and (b) there is a significant relationship between student performance on the self- study and their “overall” grade in the class. This broader test of student performance further supports the contention that the self- study is helping achieve the overall learning objectives of the course that may go beyond the standard exams. The overall grade in this study included performance on other assignments, in addi- tion to the exams, like reports and class presentations that require students to demonstrate their ability to create, process, and apply knowledge. This is somewhat different than the type of learning measured by the textbook-type exams which are more topic/ content focused. In closing, it is emphasized that this is a pilot study and the results need to be interpreted tentatively. It could be argued that the results only show correlation and not causation; yet the impli- cations are worth considering in terms of how a simulation may be used as a learning tool. The fact that the self-study is correlat- ed to a student’s overall course grade may also serve the im- portant function of being an early predictor of a student’s future performance in the course and flag those students that need addi- tional support. Further research is strongly encouraged, not only on the merits of a self-study approach to using a simulation, but on the broader issue of the pedagogical effectiveness of alterna- tive, creative/innovative, approaches for using simulations in the classroom. TABLE 4 SPEARMAN CORRELATION BETWEEN SELF-STUDY, EXAMS, AND COURSE GRADE RANK ORDER CORRELATIONS Self-Study and Exam Grades Self-Study and Overall Course Grade Spearman’s rho 0.470* 0.577** Significance (P-value) 0.037 0.008 Observations (N) 20 20 *Correlation is significant at the 0.05 level (2-tailed); ** significant at 0.01 level (2-tailed) TABLE 5 LINEAR REGRESSION OF COURSE GRADE AS A FUNCTION OF SELF-STUDY PERFORMANCE RATING Regression Statistics Adjusted R Square 0.34738 F-Statistic 9.58125 Significance: F-statistic 0.00624 Observations 20 Variables Coefficients t Stat P-value Intercept 0.242272 1.201845 0.244998 Self-Study Rating 0.669042 3.09536 0.006242 Gold , S. (2015) A Critical Examination of the “With Simula- tions it is not the wand but the magic in the magician: A Pilot Study Enhancing and Assessing Topic-Specific Learning using an Economic Simulation, Developments in Business Simulation and Experiential Learning, Volume 43, 69-75. Goosen , K. (2002) A Critical Examination of the “Experiential” Premise Underlying Business Simulation Usage, Developments in Business Simulation and Experi- ential Learning, Volume 29, 308-315 Wellington, William J., Hutchinson, David, and Faria,A.J. (2010) The Impact of Playing a Marketing Simulation Game on Perceived Decision Making Ability among Intro- ductory Marketing Students, Developments in Business Simulations and Experiential Learning, Vol. 37, 23-32. REFERENCES