With Simulations, IT IS Not the Wand but the Magic in the Magician: A Pilot Study Enhancing and Assessing Topic-Specific Student Learning Using an Economic Simulation Page 69 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 ABSTRACT This study continues a series of research papers exploring the impact on student learning associated with the use of business simulations in the classroom. The research question in this study is not whether learning occurs with a simulation, enough studies support this conclusion; but rather the pedagogical ef- fectiveness of a specific approach taken. It is not the wand, but the magic in the magician that counts. An innovative methodol- ogy is presented for both enhancing and measuring topic- specific student learning derived from a simulation game. A working example is presented to illustrate how this is done in an economic simulation game. Student surveys indicate that it is perceived by students to be an effective pedagogical approach; and student tests of learning support this conclusion. Although the example simulation game is specific to economics, the ap- proach may be applied to any business simulation. INTRODUCTION There has been an ongoing debate, since the early 1970s on the pedagogical effectiveness of simulations versus traditional methods of teaching, like lecture or case studies (Greenlaw and Wyman, 1973; Keys, 1976). After the first 25 years of research on this topic, a study by Anderson and Lawton (1995), argued the results were inconclusive, stating: “After decades of research on simulation exercises, we still cannot provide objective (versus anecdotal) sup- port for answers to questions like: Does participating in simulation exercises produce learning? If so, what kinds of learning and how do we measure it? If not, what value are simulations? Could the time spent on simulation exercises be used more effectively or pro- ductively if directed toward other pedagogies? Neither is there a consensus regarding the questions of: What knowledge, skills, and attitudes do we want students to learn from a simulation exercise?” The questions raised in this study served as a catalyst for further research. The recent studies on the learning effectiveness of simulations include three papers in 2009-2010 with more positive conclusions. Wolfe and Deloach (2009) found that the learning scores of students were positively and significantly related to the student’s performance in the simulation. Similar- ly, the paper by Gosenbud and Washbush (2010) found that students who were ranked highest in the simulation with respect to the financial indices were the ones that wrote the best reports, demonstrating a high degree of understanding of the business principles and the ability to apply them in the simulation. A thorough and comprehensive review by Wellington, Hutchinson, and Faria (2010) of the past 160 studies found that in 46.9% of the cases (75 studies) the simulation games were more effective teaching tools, as measured by common course final exams, than in courses with conventional instructional methods (typically lectures and cases). Only 16.9% of the cases (27 studies) found conventional instruction to be better; the re- maining 36.3% of the cases (58 studies) found no significant difference in the teaching performance. What may account for the different findings in these stud- ies? One possible explanation of the disparity in past results may be attributed to the diversity in the measures of learning that were used. Gosen and Washbush (2010) identified the ma- jor approaches used to measure learning, including: (a) perfor- mance in the simulation game, i.e. financial ranking; (b) exami- nations covering course learning to assess simulation learning; (c) assessments of written reports about the simulation experi- ence; (d) surveys of student perceptions of their learning; and (e) forecasting accuracy. Another insightful reason for the different findings on the learning effectiveness of simulations was implied by Goosen (2002) and needs to be emphasized, i.e. that the degree of learn- ing is highly dependent on the way in which the simulation is administered and integrated into the class by the instructor. To highlight his point, an analogy is made by Goosen (2002) with respect to the use of a “blackboard” as a teaching tool, stating: “Students cannot learn anything from an empty black- board. Similarly, transparencies are also a teaching tool. Blank transparencies are as ineffective as an emp- ty blackboard. Students can learn only if meaningful content is provided by the instructor. Blackboards, transparencies, films, videos, and yes, even simulations are delivery systems. The learning effectiveness of any pedagogical delivery system depends primarily on the skill of the user.” The key message by Goosen (2002) is that a simulation, like a blackboard, will only be an effective tool if it is used properly by the teacher, i.e. integrated into the classroom in an effective way. It is not the wand, but the magic in the magician WITH SIMULATIONS, IT IS NOT THE WAND BUT THE MAGIC IN THE MAGICIAN: PILOT STUDY ENHANCING AND ASSESSING TOPIC-SPECIFIC STUDENT LEARNING USING AN ECONOMIC SIMULATION Steven Gold Rochester Institute of Technology stevengold@saunders.rit.edu mailto:stevengold@saunders.rit.edu mailto:stevengold@saunders.rit.edu mailto:stevengold@saunders.rit.edu mailto:stevengold@saunders.rit.edu mailto:stevengold@saunders.rit.edu mailto:stevengold@saunders.rit.edu mailto:stevengold@saunders.rit.edu Page 70 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 that counts. Because of this, it is important that any study at- tempting to evaluate the teaching effectiveness of a simulation carefully detail the way it is utilized, not just if learning is oc- curring. The implication here is that the research question about the pedagogical effectiveness of simulation should change focus. The relevant question that needs to be researched is not whether simulations are effective learning tools. Enough studies have supported this hypothesis. What we need to start asking is: What are the most effective ways to use or integrate a simula- tion in a classroom? As Goosen (2002) has said, a blackboard may or may not be an effective learning tool, it depends on how it is used. The same applies to simulations with respect to peda- gogical methodology. It is this idea that has prompted this study. PURPOSE AND METHODOLOGY The primary purpose of this paper is to introduce an inno- vate approach for integrating a simulation into the classroom; and then test the learning effectiveness of this methodology. The pedagogical approach used in this study is to combine the simulation game with topic-specific simulation exercises that TOPICS EXERCISE LEARNING OBJECTIVES Getting Started Introduction To understand the game’s economic information Markets Market Equilibrium To understand how price adjusts to achieve market equilibrium using demand and supply analysis Demand Law of Demand To understand and apply the law of demand. Shifts in Demand To understand the difference between a change in demand and a change in the “quantity” demanded. Revenues Revenue Maximization To understand and predict the relationship between price, quantity demanded, and revenues. Elasticity Price Elasticity To understand the relationship between price elasticity, quantity demanded & revenues Production Short Run Production To understand the short-run relationship & to measure & apply marginal and average products Long Run Production To understand the long-run production relationship, returns to scale and the impact of plant size on labor efficiency. Cost Short Run Cost To understand, measure and apply cost concepts in the short-run. Long Run Cost To distinguish between long-run and short-run costs; understand the impact of Perfect Competition Short- Run Perfect Competition To understand how this marketplace behaves and reaches equilibrium in the short -run; and to apply economic principles to maximize profits in the short-run, and practice critical thinking skills. Long- Run Perfect Competition To understand how this marketplace behaves and reaches equilibrium in the long- run; and to be able to apply economic principles to maximize long-run profits , and to sharpen critical thinking skills. Monopoly Short- Run Monopoly To understand how this market behaves and reaches equilibrium when plant size is fixed in the short-run; to be able to apply economic principles to maximize profits in the short-run; and to provide an opportunity to reinforce critical thinking skills. Long- Run Monopoly To understand how this market behaves & reaches equilibrium when plant size may change in the long-run; to be able to apply economic principles to maximize profits in the long-run; and to sharpen critical thinking skills. Monopolistic Competition Short- Run Monopolistic Competition To understand how this marketplace behaves and reaches equilibrium in the short -run; to learn to apply economic principles to maximize profits in the short run; Long -Run Monopolistic Competition To understand how this marketplace behaves and reaches equilibrium in the long- run; to learn to apply economic concepts to maximize profits in the long-run; and to sharpen critical thinking skills. Oligopoly EXHIBIT 1 LIST OF SIMULATION EXERCISES AND LEARNING OBJECTIVES Page 71 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 utilize the simulation game as the frame of reference. These topic-specific simulation exercises are very different than simu- lation games. In the simulation exercises students observe what is happening in pre-set games and then are required to an- swer a series of multiple choice questions or problems to test their knowledge of specific topics in the course. The innovation is achieved by uniquely combining simulation “exercises” with simulation “games”. The significant advantage of this method- ology is that it is designed to directly measure the learning asso- ciated with the actual content (theories) embodied in the simula- tions that are relevant to the course. More details on the design of the simulation exercises follow in the next section. Seventeen different topic-specific simulation exercises were provided as part of an economic simulation package called “Beat the Market: An Interactive Microeconomic Simulation Game” (www.btmgame.com) and were used in this study to test the effectiveness on student learning. The exercises were as- signed in three foundation economics courses during the past three semesters. About half of the exercises were required to be completed by each student individually during the course, while at the same time they were participating in a class “team-based” simulation competition. The student exercises and team simula- tion competition were assigned over a 9 week period of time. DESIGN OF SIMULATION EXERCISES The simulation exercises were designed to enhance and test student learning. Exhibit 1 displays a list of the 17 topic- specific exercises along with their associated learning objec- tives. As an example, Exhibit 2 shows the directions the students are given for the Price Elasticity exercise. All exercises have the same general directions. Near the top of Exhibit 2, students are shown the due date. Next, the “exercise information” indicates that students will be able to see their grades and answers to the exercise after they submit their answers; and that they can re- peat the exercise up to 3 times. The instructor has the option of changing: (i) whether students see their grade and the answers before or after the due date; and (ii) the number of times stu- dents may repeat the exercise. The instructions to “Play the Exercise Game” in Exhibit 2 are important. Students must enter the decisions directed in the questions that are specified in the exercise (see Exhibit 3) and then answer the questions (fill in the blank or multiple-choice) based on these decisions. With most of the questions, the stu- dent does “not” have the discretion to make up their own deci- sions when they play the game for the exercise. The important EXHIBIT 2 PRICE ELASTICITY EXERCISE – DIRECTIONS TO STUDENTS http://www.btmgame.com Page 72 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 point is that in all the exercises, it is not the game performance in terms of profits (etc.) that counts. Rather, students are graded on the number of correct responses to the questions. If a student made a mistake entering a decision in the exer- cise game, the student may press the RESET button to “Reset a Game”. For example in Exhibit 3 question 1, the student is in- structed to lower price by $2. If, by mistake, they lowered price by less or more than this, then pressing RESET will start the game over again. After answering all questions, the student presses SUBMIT and will see their grade and the correct answers. If the student does not like their grade, they may press REPEAT to re-do the exercise. In this example, they may re-do the exercise two more times after the first try. The instructor has the option to select the number of repeats. In our study we kept the number of re- peats allowed at 3. When an exercise is “repeated”, students will get a similar but different game in terms of demand, costs, and market condi- tions. In this case the answers to the questions will change, that is, they cannot just copy the correct answers from the prior ex- ercise or from another student. Even the first time an exercise is played, the game will be different for every student. So if the exercise is assigned as homework, one student cannot copy the answers of another student. There is also a risk to the student of repeating an exercise. The grade a student receives is the perfor- mance on the last exercise taken. For instance, if a student earns an 81% on the first exercise, a 90% on the second attempt, and an 85% on the third repeat, the students grade will be 85%. But if the student stopped after the second attempt, the grade as- signed would be a 90%. Exhibit 3 illustrates the instructions in the Price Elasticity Exercise for the beginning questions 1 and 2; and then the end- ing questions 13 & 14. The questions include “fill in the blank” and “multiple-choice” and provide a mix of questions appropri- ate to the course. In question 1, students must follow the direc- tion to lower price by $2 and the answers in questions 1 and 2 depend on the pricing decision. Question 13 and 14 depend on the price elasticity calculated in question 11 (not shown here). If the price elasticity calculated in question 11 is wrong, the stu- dent can still get questions 13 and 14 correct, if they properly apply the assumed price elasticity value from question 11. Re- member that the correct answers will be different for each stu- dent since the game is algorithmically driven with different pa- rameter values each time a new exercise and game is accessed. ASSESSMENT OF LEARNING EFFECTIVENESS To test the effectiveness of the exercises in terms of student learning, the grades of the exercises were recorded and a survey of student perceptions of the exercises were done in three foun- dation economics courses. The exercises were assigned to be completed by each student individually, along with participating EXHIBIT 3 PRICE ELASTICITY EXERCISE QUESTIONS 1-2 & 13-14 Page 73 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 in a “team” based simulation game. The student exercises and team simulation game were assigned over a 9 week period of time. After this was completed, the students were given a stand- ard class exam that covered the course content up to this point. The performance on this exam was then compared to other ex- ams that were given without the use of a simulation. The grades received on the exercises are presented in Ex- hibit 4. The average grade for the 66 students completing the exercises was 92.4% with a standard deviation of 5.9%. The grade results were similar between the three classes with no significant difference. The high average grades imply that the student’s learned the topic-specific economic concepts meas- ured by these exercises. The students were also surveyed on their own perceptions of learning from the simulation exercises; which has been a common approach used to measure learning in past studies (Comer and Nichols, 1996; and Herz and Merz, 1998). In our study, students were asked only two questions about the learn- ing effectiveness of the simulation exercises. Out of the 66 stu- dents that were surveyed only 6 did not reply. The survey ques- tions and results are presented in Exhibit 5. The first question asked if the exercises helped students “understand” the econom- ic concepts. The second question asked if it helped students learn how to “measure and apply” the economic concepts. We viewed the first question as a more theoretical understanding of the course content, as opposed to the ability to use/apply the theoretical concepts. In 100% of the responses the students agreed or strongly agreed that it helped them “understand” the economic concepts. In terms of helping to measure and apply the economic concepts, 89% agreed or strongly agreed that it did so. To ascertain the impact of the simulation on the “standard” class exams, two exams were administered in each of the three courses used in this study. Exam 1, given as a midterm test cov- ered microeconomic content from the textbook that was also reinforced by the simulation exercises and games. Exam 2, giv- en at the end of the course as a final test covered macroeconom- ic content from the textbook that was not related to the simula- tion. In this part of the course, no simulation was used, just the standard lecture with weekly homework assignments, like quiz- zes or reports. The student scores on exams 1 and 2 are report- ed in Exhibit 6. The students scored consistently higher on ex- am 1 compared to exam 2, by about 4 to 5 points, in all three classes. The differences in the mean grades were statistically significant at the 0.05 level, using a one-tailed t-test, in two of the three classes. One possible criticism of this analysis may be that the difference in the exam grades could be attributed to the type and complexity of the content that is covered between microeconomics versus macroeco- nomics. Although this may be true, the content is still foundation eco- nomics and the same textbook was used in this study to teach both microeconomics and macroeconomics, which controls the level of rigor. As further support, a study by Terry and Galchus (2003) pre- sented data showing that there was no statistically significant difference in student course grades between microeconomics and macroeconomics; and no studies to the contrary were found in the literature. SUMMARY AND CONCLUSIONS The intriguing question of if and what students learn using a simulation has been debated in the educational literature for many years with a diversity of findings with respect to its effec- tiveness. An insightful reason for the inconsistent findings was pointed out by Goosen (2002), and later by Gosenpud and Washbush (2010), that the learning effectiveness is highly de- pendent on how the simulation is integrated into the classroom by the instructor. So it is important to understand exactly how the simulation is utilized in the classroom before making con- clusion on its learning value. Carrying forward with this idea, our study focuses on de- tailing and measuring the effectiveness of a particular pedagogi- cal approach for integrating a simulation into the classroom. The methodology tested in this study is to use topic-specific simulation exercises to enhance the course learning objectives; and then couple this with the use of a comprehensive simulation EXHIBIT 5 SURVEY OF STUDENT OPINION ABOUT SIMULATION EXERCISES & GAMES Survey Questions (60 responses) Strongly Agree Agree Neutral Disagree Strongly Disagree Simulation Exercises helped me understand the economic concepts in the class 66.67% 33.33% 0% 0% 0% Simulation Exercises helped me learn how to measure & apply the economic concepts 66.67% 22.22% 11.11% 0% 0% EXHIBIT 4 SIMULATION EXERCISE TEST RESULTS OF STUDENTS ECONOMICS COURSES (3) RESULTS Number of Students 66 Average Simulation Exercise Grade 92.4% Standard Deviation 5.9% Page 74 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 that covers a wide range of course learning objectives. The results show that (a) test scores on the topic-specific simulation exercises are high, typically over 90%, indicating the exercises are helping students grasp the specific learning objec- tives; (b) survey results show that students perceive that the topic-specific simulation exercises help them understand the course concepts; and (c) grades on standard exams are higher when the simulation exercises were used compared to the exam grades when simulations were not used, just standard lectures and conventional homework (like problems and reports). These results support the hypothesis that simulations are effective pedagogical tools if used in the manner described in this study. But it is also important to be able to explain and sup- port, theoretically, the possible reasons for the positive findings. Two reasons follow. First, the use of topic-specific simulation exercises employs a “step-wise” learning methodology. Not everything is covered at once, as is typically done by the use of a comprehensive sim- ulation that requires the student to draw upon and apply all as- pects of the course content, including critical thinking skills, all at once. With our methodology, the student is given a support- ing framework to learn individual components of the learning objectives of the course in a step-by-step fashioon. Second, the use of the comprehensive simulation game, following the topic-specific simulation exercises, increases the likelihood the student will understand the synergistic relation- ships between the specific topics in the course, and how to bet- ter apply them in a more holistic fashion. For example, lowering price and selling more goods, has an impact on both revenues and costs of the firm. By providing simulation exercises that first focuses on the relationships between price, sales units, and revenues; and then a set of exercises on how sales impacts pro- duction and costs, will help students better perceive the syner- gistic relationship between the two factors when done together in a more comprehensive simulation game. . Yet, it should be emphasized that this is a pilot study and the results should be interpreted tentatively. First it is based on data from only 66 students. Second, the issue of what is the appropriate measure of learning with respect to the use of the simulations needs to be more thoroughly addressed. The issue of what type of learning we want to achieve and measure is still a relevant but challenging question to answer. For example, the study by Wolfe and Deloach (2009) define learning as the abil- ity to create, process, and apply knowledge to useful ends. This is somewhat different than the type of learning measured by the exercises in this study which are more topic/content focused; including the ability to apply knowledge but not create knowledge. More research on how to measure different types of learning in simulatoins is needed. Finally, a strong case is made to change the past research focus of the learning derived from simulations to the question of the pedagogical effectiveness of the specific approach taken to use the simulation by the instructor. As Goosen (2002) contends “it is not the wand, but the magic in the magician”! REFERENCES Anderson, P. and Lawton, L. (1992). A survey of methods used for evaluating on student performance on business simulations. Simulation & Gaming, 23, 490-498. Anderson, P. & Lawton, L. (1995). The problem of determining an “individual simulation’s validity as an assessment tool. Development in Business Simulations & Experi- ential Exercises, 22, 43-48. Comer, L.B. and Nichols, J.A.F. (1996). Simulation as an aid to learning: How does participation influence the pro- cess? Developments in Business Simulation & Experi- ential Exercises, 23, 8. The Bernie Keys Library. Goosen , K. (2002) A Critical Examination of the “Experiential” Premis Underlying Business Simulation Usage, Developments in Business Simulation and Experiential Learning, Volume 29, 308-315 Gosenpud, Jerry and Washbush, John (2010). The Relationship between Learning and Performance in a Total Enter- prise Simulation: Revisited and New Data, Develop- ments in Business Simulation & Experiential Learning, 37, 95-98. Reprinted in The Bernie Keys Library, 13th edition. Greenlaw, P.S. and Wyman, F.P. (1973). The Teaching Effec- tiveness of Games in Collegiate Business Courses. Simulation & Games, 4, 259-294. Herz B. and Merz (1998) Experiential learning and the effec- tiveness of economic simulation games. Simulation & Gaming. Vol. 29. 238-250. Keys, B. (1976). A Review of Learning Research in Business Gaming. Developments in Business Simulation & Ex- periential Exercises, 3, 25-32. Patz, A (2000), One more time: overall dominance in total en- terprise simulation performance.Development in Busi- ness Simulation & Experiential Exercises, 27, 254- 257. EXHIBIT 6 EXAM PERFORMANCE WITH AND WITHOUT SIMULATION EXERCISES & GAMES RESULTS Class 1 Grades (33 students) Class 2 Grades (14 students) Class 3 Grades (19 students) Mean Std.Dev Mean Std.Dev Mean Std.Dev Exam 1 with simulation 90.08 9.30 88.56 7.90 85.84 9.18 Exam 2 without simulation 85.57 9.26 84.55 8.82 81.03 5.68 Difference in Mean Grade 4.51 4.01 4.81 T-test: difference in means 1.97* 1.27 1.94* *Statistically significant difference in means at the 0.05 level; one tail test Page 75 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 Specht, Linda B. and Sandlin, Petrea K. (1990). “A Comparison of the Effects of Experiential Learning Activities and Traditional Lecture Classes”, Developments In Busi- ness Simulation & Experiential Exercises, Volume 17, 214. Terry, A. and Galchus, K. (2003), Does Macro/Micro Course Sequencing Affect Student Performance in Prinicples of Economics Courses? Journal of Economics and Fi- nance Education, volume 2, number 2, Winter 2003, 30-37. Wolfe, J. and Deloach, G. (2009). Individual and organizational learning in a top management game. Developments in Business Simulation and Experiential Learning, Vol. 36, 231-237. The Bernie Keys Library. Wellington, W. J. and Faria, A. J. (1992). An investigation of the relationship between team cohesion, player atti- tude, performance attitude on player performance. De- velopments in Business Simulation & Experiential Ex- ercises, Vol. 19, 184 189. The Bernie Keys Library. Washbush and Gosen (2001). Learning in total enterprise simu- lations, Simulation & Gaming, Vol. 32, 3, 281-296. Washbush and Gosen (2002), Total Enterprise simulation Learning Compared to Traditional Learning in the Business Policy Course, Developments in Business Simulation and Experiential Learning, Volume 29, 281 -286. The Bernie Keys Library. Wellington, William J., Hutchinson, David, and Faria,A.J. (2010) The Impact of Playing a Marketing Simulation Game on Perceived Decision Making Ability among Introductory Marketing Students, Developments in Business Simulations and Experiential Learning, Vol. 37, 23-32.