Some Recommendations for Researching Learning from Playing a Simulation Page 206 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 ABSTRACT This paper attempts to help simulation scholars understand why most of the research attempting to show that simulation players learn from playing fails. It recommends research designs that are simpler than those historically used. It recommends that independent variables reflect how simulation play is organized and what aspects of a business are emphasized in given simulation. It also recommends that dependent variables reflect learning from the simulation itself and also recommends research populations from multiple universities. THE PROBLEM Much has been written about learning in the simulation. Most who teach with simulations believe that simulations are valid, and Faria and Wellington (2004), Crocco, Offenholle, and Hernandez (2016), and Wilson et al. (2009) are among those that contend that students do learn from them. But the evidence supporting that contention is not convincing. In most simulations, the difference between good and not good performance is clear. The trouble is that while simulation scholars contend that that simulation players who perform the best learn the most, there are no studies that clearly show that. There may be hundreds of studies that cover the topic of players learning from the business simulation experience. A review of the citations from Anderson and Lawton's paper on business simulations and cognitive learning (2009) shows that 35 of their 65 citations covered learning from playing a simulation. Twelve of the 21 citations from Washbush and Gosen (2001) dealt with whether or not students learned from playing a simulation, and only four of those 12 overlapped with the Anderson and Lawton list. That means that in those two papers alone, there are citations from 47 articles that focused on learning from playing a simulation. And as argued above, none have shown clearly that the best simulation players learn the most or more generally that simulation players learn from playing. WHY DO WE LACK STUDIES THAT SHOW THAT SIMULATION PLAYERS WHO PERFORM THE BEST LEARN THE MOST The answer is that much of the simulation-learning-related research literature features the complexities of learning to the detriment of clear results. For most fields, though, what we know comes from common sense observations of simple experiments comparing obviously different groups, with results that are likely to be obvious and predictable. If you had a basketball game between a random sample of 5' 8'' men on one team and a random sample of 6' 3" men on the other, the taller men would win most of the time and height would the reason. There are definitive studies that show that health related outcomes are significantly better for babies that are breast fed for their first six months than for formula fed babies, as long as other factors are held constant (babycenter.com, 2015). From the simulation literature, Wolfe (2016) showed that students game players who spent the most time preparing to make a greater variety simulation decisions and did so through the entire game performed better, as measured by profits, on the simulation than students who spent less time, were inconsistent across time and focused on fewer decisions. In contrast to the above examples where independent variables are concrete, many of the studies that focus on learning from playing simulations include variables that complicate matters. For example, instead of simply trying to determine whether players learn from playing a simulation, scholars such as Anderson and Lawton (2009), Hsu (1989), and Wolfe (1985) seem to want to understand the learning process and differentiate some kinds of learning from others in addition to determining whether or not simulation players are simply learning. Adding learning process variables complicates efforts to find out whether or not simulation players are learning from their experience. For example, Anderson and Lawton's study (2009) focused on Bloom's Taxonomy (Bloom, Englehart, Furst, Hill, and Krathwohl, 1959) and distinguished between learning complexity levels using the taxonomy to help distinguish low from high levels of learning. Understanding learning and distinguishing between learning complexity levels is valuable, and the Bloom model is still widely cited. However, the learning process is extremely complicated, and trying to understand its complexity in the context of one learning exercise (playing a business simulation) is very difficult, and if our goal is to prove that those who play simulations learn from playing, counter-productive Some Recommendations for Researching Learning from Playing a Simulation Jerry Gosenpud University of Wisconsin-Whitewater gosenpuj@uww.edu http://www.babycenter.com mailto:gosenpuj@uww.edu Page 207 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 WHY SIMULATIONS ARE EXCELLENT WAYS TO TEACH MANAGEMENT This paper presumes that active learning is the best way to learn. Learning to manage businesses is difficult, and learners need experience in managing in order to learn to manage. That does not happen with lectures and does not happen by knowing theories. Reading and analyzing cases might help one learn to manage, but understanding cases is probably more valuable if learners have first hand experience in managing (or at least observing managers) and cases were found to be inferior to simulations in improving self-efficacy in Strategic Management courses (Thompson and Dass (2000). Does learning ever take place for college management students? Yes, with internships and with on the job training, both usually undertaken outside the actual classroom. But learning inside the classroom is unlikely to take place without some active learning activity. What is a good in-class active learning procedure for college undergraduates to learn how to strategize and manage? The computerized business simulation is one answer. It is active learning in that players analyze information and make decisions that help or hinder business growth, profitability or other indices of business success. An advocate of simulations would expect that simulation players who perform well in a simulation will learn more than players who perform poorly. It is therefore desirable that researchers perform studies that attempt to verify this. HOW DO WE DESIGN STUDIES THAT PROVE THAT SIMULATION PLAYERS LEARN FROM PLAYING? This paper recommends a research design that will help us prove that simulation players learn from their playing experience. Research designs should be easy to follow. Independent variables should be easy to understand and cover easy-to- identify features of the simulation experience. Dependent variables should be a measure of learning. INDEPENDENT VARIABLES We can do experiments with simulations that will likely attain results as clear as the above hypothetical basketball match or the breast-feeding study. Below are some examples of simulation-related independent variables. They are proposed to help show that learning does take place by playing a simulation. The first three of these examples in section A were proposed by Faria (1986), and examples in section B were proposed by Wolfe (2016) or borrowed from Micromatic, version 4 (Oaktreesim.com micromatic). These independent variables are simple rather than complicated and are easy to understand. A. Features of the experience faced by students (Faria, 1986) A1. Amount of explanation given to students. Some students received detailed explanations (or training) before the start of the simulation, while others did not receive such information. A2. Team size. The students were divided into groups of three (51 teams), four (35 teams), and five (49 teams). A3. Grade weighting. The game results counted for 40% of the final grade for about half the students and 20% for the other half. Learning was not measured, but other dependent variables were. The teams of 3 outperformed the other two groups, but teams of 5 out performed teams of 4. High explanation teams worked together more cohesively than teams that were given no game explanation. Surprisingly those whose game score was 40% of the grade did not spend more time on the game than those whose game score was worth 20% of the grade. B. Emphasis on what business-related activity leads to success for a given player Games are played with contrasting emphases as to what leads to success. Many games reward multiple emphases, so for research purposes, researchers can manipulate reward schemes so some emphases yield a higher financial outcomes and grade than others. For example, some students will be assigned to play games that emphasize and reward sales volume, while other students will be exposed to games with an emphasis on return on sales. Still other students will play games rewarding e.g., investments in technology, capacity utilization, or cost of goods sold (Oaktreesim.com/ micromatic; Wolfe 2016) What leads to success in a given game defines the independent variables of the research for that game's student subjects. So if sales volume is rewarded in a given game, then one of the independent variables of a particular study would be whether or not (or the degree to which) sales volume is rewarded, and it would be hypothesized that those who played games in which sales volume is rewarded will learn more about the impact of sales volume on game performance than those playing games with other game goals (e.g., the impact of funding automation). The purpose of Wolfe's paper (Wolfe, 2016) was to propose using simulations as tools to verify assurance of learning in university business schools. Wolfe's simulation, The Global Business Game: World Edition is complicated with seven types of Page 208 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 decisions: scanning, accounting, finance, marketing, logistics, operations, and strategic alliances. The game is set in a mature industry suggesting keys to success such as increasing sales, low overhead, high capacity utilization, reducing cost of goods sold, and increasing automation levels. Wolfe hypothesized that the most successful firms (all of his simulation players played in teams) would strategize and implement decisions consistent industry conditions. The results were consistent with his hypotheses. No one has tried to replicate these two studies, presumably in part because of the large number of subjects required. Faria taught at Windsor University where all students in the capstone business course were required to play the same simulation at the same time. At Windsor, more than 500 students participated. Wolfe performed his study at a western European university, where 836 students participated, all playing the same game (The Global Business Game: World Edition) under the same industry conditions. Most researchers interested in simulations do not work in universities where there are large enough numbers of students taking a simulation course at the same time to replicate the size of these two studies, and it is important for sample size to be large enough to create statistically significant results. So scholars from multiple universities should work together and create sample sizes that are large enough so the statistical significance is feasible. This is difficult and requires a lot of work for researchers. Doing so is possible, though, and worth the effort if we want to prove beyond doubt that simulation play leads to relevant learning. DEPENDENT VARIABLES Though out this paper, I've argued that it is important to prove that learning takes place from playing a simulation and pointed out that there are few if any simulation related studies where an attempt was made to create a separate variable that clearly represents learning from playing a simulation. This is important because it s very valuable to know if a simulation is effective in teaching what it was designed to teach. In contrast, if researchers try to prove that simulations are valid using a more general learning-related variable, such as course learning, it will be harder to prove that students learn from the simulation because other course events can modify the impact of the learning originated from the simulation. It should be noted that a study attempting to show that learning from the simulation does take place has been undertaken. Wolfe and Roberts (1985-6) did a study featuring a multiple-choice exam consisting of 23 questions testing whether students learned from trying the do well in the simulation. Twelve of the questions covered each of the simulation company’s functional areas, i.e., marketing, finance, production, and management and eleven covering the integration of two functional areas. For the present paper’s purposes, student answers to test questions were significantly more correct after playing the simulation than before for three of the twelve functional areas and for one of the of the questions integrating two functional areas. The recommendation here is that simulation scholars collaborate in determining learning goals from playing a simulation, create ways to test student players as to their understanding of the business related principles that the game they have played teaches. Simulation scholars should also design the tests. If it is important to assess the effectiveness of the simulation, then the dependent variable should reflect what authors and teachers expect students to learn as they attempt to do well in the simulation. In other words, simulation players should take tests in part to help us discover whether they have learned from simulation play. Tests should be multiple choice or short answer, so whether an answer is correct is not a subjective decision. Exam scores should be part of the student’s simulation grade. There are many principles that games teach. Some games reward training and treating employees well. Others reward investing in technology, others reward return on sales. Still others may reward keeping debt low, and there are situations where it is appropriate to keep debt reasonably high. Of course, many simulations reward combinations of strategies, and of course there are many other strategies that simulations encourage that are not mentioned here. If a game has a particular focus, then tests measuring student competence on that game should cover that focus. The prediction is that those playing games rewarding a particular focus, for example return on sales, will score better on an exam covering the consequences of ROS than those playing games focusing on something else, for example, training. NOTES: It should be noted Wolfe (2016 and 2018) argues that it is more important that simulation competence contribute to course learning and in general to overall business success than it does to simulation learning. And he seeks to show in his 2018 article a relationship between simulation success and overall career success. Ironically, while Wolfe in 2016 and 2018 argues the importance simulation competence contributing course and career success, Wolfe and Roberts in 1985-6 were among the few scholars who used exam scores reflecting learning from the simulation itself as a key dependent variable. In that study the players who learned the most from playing were more consistently active and thorough as they approached playing the simulation. I would argue that there’s value in using any one of the of the discussed dependent variables, learning from the actual simulation, course mastery, and career success. Page 209 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 Anderson, P. H. and Lawton L. (2009). Business simulations ad cognitive learning: development, desires, and future directions: Simulation and Gaming: 40 (2), 193-216 Baby Center,com https://www.babycenter.com/0_how- breastfeeding-benefits-you-and-your-baby_8910.bc Bloom, B. S., Englehart, M.D., Furst, E.D., Hill, W. H., and Krathwohl, D.R, (1959). Taxonomy of Educational Objectives: e Classification of Educational Goals. Handbook 1, The Cognitive Domain New York: David McKa Inc. Crocco, Francesco, Offenholley, Kathleen, Hernandez, C. (2016). A proof-of-concept study of game-based learning in higher education. Simulation and Gaming, 4c7, 403-422 Faria A. J. (1986) A test of student performance and attitudes under varying game conditions. Developments in Business Simulation & Experiential Learning, 13, 70- 78 Faria, A. J and Wellington W. J. (2004). A survey of simulation of game users, former users and never users. Simulation & Gaming, 35, 178-207 The Global Business Game: World Edition Hsu, E. (1989) Role-event gaming in management education: A conceptual framework and review. Simulation and Games, 20, 409-438 Oaktreesim.com micromatic http://oaktreesim.com/MMV4/ Thompson, George H and Dass, Parshotam (2000) Improving students’ self-efficacy in strategic management: the relative impact of cases and simulations. Simulation and Gameing, vol.31 (1) 22-41 Washbush, J. and Gosen, J. (2001), An exploration of game derived learning in total enterprise simulations. Simulation & Gaming, vol. 32 281-296 Wilson W. L., Bedwell E. H., Lazzara E.,Burke,S., Estock, J.L., Orvis, A., and Cook, C .(2009), Relationships between game attributes and learning outcomes, Simulation & Gaming, vol. 40 (2) 217-266 Wolfe, J. (1985), The teaching effectiveness of games in collegiate business courses: a 1973-1983 update. Simulation and Games, 16 (3) 251-288 Wolfe, J. (2016) Assuring business school learning with games. Simulation and Gaming, 47 (1), 1-22 Wolfe. J. (2018) Who Learns in a Business Game: An Objective Single Player Examination. Development in Business Simulation and Experiential Exercises vol. 45 Wolfe, J. and Roberts C. R. (1985-6). The Effect of Different Grade Weigjhts on Business Game Learning. J. of Educational Technology Systems, Vol 14 (4) 307-316 REFERENCES https://www.babycenter.com/0_how-breastfeeding-benefits-you-and-your-baby_8910.bc https://www.babycenter.com/0_how-breastfeeding-benefits-you-and-your-baby_8910.bc http://journals.sagepub.com/author/Crocco%2C+Francesco http://journals.sagepub.com/author/Offenholley,+Kathleen http://journals.sagepub.com/author/Hernandez%2C+Carlos