How Students “Play” Business Simulations and What They Learn: The Preliminary Report Page 89 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 ABSTRACT This paper is the initial report on a large-scale study of how business students participate in business simulations and, in general, what they learn by participating in them. Many studies have been conducted attempting to measure the amount of learning that takes place when students participate in business simulations to little avail. The vast majority of these studies have attempted to link individual/team performance to learning, but that faces at least three major stumbling blocks; 1) simulations, in general, are played by teams, and one cannot determine which team member did what tasks. 2) The learning that takes place may be non-cognitive or training, not cognitive learning. And 3) Most of these studies use data from students who reside in one university, are in a class or classes taught by one instructor and are “playing” a single business simulation. This study is an attempt to determine differences in gender, US students vs. international students, simulations played by teams vs. individual players, and the final performance of the simulated firms. Each of these measures requires more data than is typical in studies using a single class or by the instructor or for a single simulation. LOOKING AT PAST STUDIES ABSEL members, as well as many others, have published many studies of comparing pseudo-learning measures and simulation performance to little avail (Anderson and Lawton, 2009). Wolfe and Willis (2017) claim, “After over 60 years of business game usage in higher education, there is still no objective evidence that they teach a course’s subject matter” (quote from the abstract). In studying the performance of teams participating in the 1974 Emory University Intercollegiate Business Game (I.B.G.), Armenakis et al. (1974) studied seven research questions including, “Do successful teams employ more quantitative methods than unsuccessful ones?” Other research questions related to prior experience with I.B.G., experience with any business game, time spent making decisions, team composition, team size, and team member role rotation over the course of playing the game. Other than participants perceiving greater learning when their role rotated on the team during the simulation, they found no statistically significant differences. They concluded, “the fact that there were no significant differences between successful and unsuccessful teams to be a significant finding” (Armenakis et al., 1974, p. 277) and called for more research in this area. In a study using a project management simulation, Szot (2013) built on several techniques found in the literature for measuring the effectiveness of simulation game experiential learning to examine the use of the simulation game as a capstone learning experience in a project management, graduate degree program. Using mixed method techniques, he found favorable attitudes towards the overall experience, significant increases in student perceptions of knowledge and the ability to apply that knowledge, significant increases in attitudes towards their team and teamwork, and no significant increase in an already favorable attitude towards simulation games. Categorization of narrative response data supported these findings and, taken together, both supported the conclusion that it appears students found the project management simulation game to be a valuable experience and an effective use of class time. We have known for a long time of the lack of evidence of the link of causality between winning a simulation and learning. See the following authors to review this evidence (Anderson & Lawton, 2007; Anderson, Lawton, & Wellington, 2008; Chin, Dukes, & Gamson, 2009; Dukes & Seidner, 1978; Gosenpud, 1990; Taylor & Walford, 1978; Teach & Patel 2007). In spite of the lack of evidence, when business simulations have been used, the simulated firm’s earnings are heavily weighted as the measure of success (Teach and Patel, 2007). Profits are then used to reward the students who “played” the business simulation by giving the best grade to the team that won. Why not? That is the measure of use in the business world. But, the business school’s classrooms are not the business world, its purpose is to educate future business leaders. In contrast to this approach, Szot (2016) bases his grading on his observations of the students’ approach to playing the game and on the quality of their reflections about the experience including their answers to the question, “What did you learn about project management from participating in the simulation?” How students “play” business simulations and what they learn: The preliminary report Richard Teach Georgia Institute of Technology richard.teach@scheller.gatech.edu James Szot The University of Texas at Dallas jimszot@utdallas.edu mailto:richard.teach@scheller.gatech.edu Page 90 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 THIS STUDY The intent of this study is to study differences in learning between groups of students that are almost impossible when studying a single business simulation, a single instructor’s class or classes, or a single university. We are attempting to generate data from many business classes taught in many universities. This study is the initial report from a pilot study using four different classes in a single university. THE PILOT SURVEY Students were also asked to evaluate their agreement with statements beginning with “After ‘playing’ the simulation, I felt I had the ability to…” followed by the same sixteen skills. Rather than indicate the level of agreement, students could also respond “not applicable” if the simulation did not include the subject provided in the question. On both prior-coursework preparations and after-simulation abilities, the students recorded much higher scores and lower standard errors than anticipated. These phenomena may be a function of the specific university selected for the pilot study. The participants recorded the name of the simulation. They also reported their ages; the academic nature of the courses in which the “played” the business simulation; their gender, their simulated firm’s ending position after the simulation was over; the number of rounds played; if they “played” the simulation as a single individual or as a team member, and the country in which they graduated from high school. This last question was to differentiate students from the USA from international students. After the pilot study, we expect to have our survey completed by many students at many universities. We tried running a series of discriminate analyses, but these did not produce useful results due to insufficient sample sizes. We also ran a set of 2x2 or 2x3 crosstabs. In this process, we found the respondents seldom selected any one of the “disagree” statements. As a result, we combined the three scale points assigned to Disagree (Strongly, no adjective and Slightly) into a single “Disagree” category to eliminate very small expected values in the crosstabs. We also attempted to factor analyze the 16 Likert-like questions regarding the students’ preparations and the 16 Likert-like questions regarding their perceived post-simulation ability. In both cases, we obtained a highly significant Bartlett’s Test (p < 0.0005). However, the Kaiser-Myer-Olkin Measure of sample adequacy was only 0.611for both tests. Thus, we need a larger sample to be able to create adequate factors. THE RESULTS OF THE PILOT STUDY Cross-tabs As this was a pilot study with a small sample size, and the sample size is an important issue in determining significance, we will report variables with p-values often greater than studies normally use. Tables 3 through 12 show the p-values for the differences between specific groupings of students using the variables from Table 2. Table 3 reports the gender differences. Table 4 shows the differences when business simulations have differences in the number of rounds played. For the question, The Number of Rounds Played, very few respondents reporting “3 or less” or “4 to 6.” so we consolidated the number-of- TABLE 1. THE LEARNING VARIABLES 1) Set goals, 2) Make competitive decisions. 3) Differentiate important information from unimportant information. 4) Work well in teams. See Hall 2012) 5) Do marginal analysis 6) Work well under uncertainty. 7) Forecast outcomes 8) Analyze reports and financial results. 9) Create budgets. 10) Understand the interaction of two or more decision variables. 12) Anticipate competitive reactions to our/my decision. 13) Assess risk. 14) Consider possible competitors when making our own firm’s competitive decisions. 15) Be innovative. 16) Be creative. Page 91 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 rounds responses in a way resulted in no expected values of less than five and left three groups: Less than 10 rounds; 11 to 12 rounds; and greater than 12 rounds. Having participants report the county in which they graduated from high school was used as a surrogate to identify international students. International students and US students differed on the five variables shown in Table 5. For this question, “What country did you graduate from high school,” we categorized into two groups: students from the USA (plus one Canadian respondent on the assumption that Canadian culture is very similar to the US culture) and all other nationals. TABLE 2 MEANS AND STANDARD ERRORS OF THE ENTIRE SAMPLE BOTH ADEQUATELY PREPARED AND ABILITY AFTER PLAYING VARIABLES Adequately Prepared Ability After Playing Variable Name Mean Std. Err Mean Std. Err. 1) Set goals, determine actions to achieve the goals, and mobilize resources to execute the actions. ……….. 5.36 0.062 5.07 0.085 2) Make competitive decisions. …..…………………….... 5.29 0.067 5.09 0.083 3) Differentiate important information from unimportant information. …………………………………………… 5.26 0.064 5.08 0.076 4) Work well in teams. …………...……………………… 5.38 0.071 4.9 0.128 5) Do marginal analysis. ……………...………………….. 4.9 0.07 4.69 0.087 6) Work well under uncertainty. …………...…………….. 5.01 0.075 4.98 0.078 7) Forecast outcomes, such as cash flows, units of ending inventory, unit demand, etc. ……………………………. 5..26 0.071 4.9 0.085 8) Analyze reports and financial results. …………..…….. 5.3 0.067 5.06 0.075 9) Create budgets. …………………………………….….. 5.07 0.72 4.76 0.092 10) Understand the interaction of two or more decision variables. …………………………………….………… 5.22 0.065 5.08 0.071 11) Analyze Quality Control measures. …………...………. 5.02 0.071 4.73 0.096 12) Anticipate competitive reactions to our/my decision. 5.07 0.073 4.06 0.094 13) Assess risk. ………..………………………………….. 5.05 0.071 4.96 0.084 14) Consider possible competitors when making our own firm’s competitive decisions. ………..………….. 5.15 0.069 4.86 0.092 15) Be innovative. …………………………..……………. 5.25 0.071 4.77 0.11 16) Be creative. ………………………………..…………. 5.19 0.077 4.71 0.109 TABLE 3 DIFFERENCES BETWEEN MEN AND WOMEN Variable name “p” value Your coursework completed prior to the simulation adequately prepared you to: work well in teams. ………………………………………………………….…. “p” = 0.104 TABLE 4 DIFFERENCES IN ROUNDS PLAYED Variable name “p” value Your coursework completed prior to the simulation adequately prepared you to: set goals, determine actions to achieve the goals, and mobilize resources to execute actions. ………………………………………………….. “p” = 0.091 work under uncertainty. …………………………………………………………………………….…… “p” = 0.043 Page 92 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 The next set of differences involved how the teams placed after the simulation was over. We asked the teams, “How many rounds or periods did your simulations run?” Their responses were limited to 3 or less; 4 to 6; 7 to 9; 10 to 12; and 13 or more. Very few students responded as having finished last or next to last. It could be respondents fibbed, or those that finished last or next to last did not complete the survey. We selected two groups of tests for differences: those who reported finishing first and those finishing in the middle of the pack. Table 6 lists the variables that discriminated between these two groups. The survey asked the students: “Was the simulation team-based or did you complete it as an individual?” They responded to this question by selecting either “it was a single-player based simulation” or “it was a team-based simulation.” Table 7 shows the differences between these two groups. Participating in a business simulation as a team is quite different than when one plays as an individual decision maker. The expansion of this survey will explore these and other differences more clearly. The previous analysis used two-way crosstabs, a non-metric, distribution-free analysis that does not allow determination of the direction of the differences. Comparing Means To assess the direction of the differences, we compared mean values of the variables looking for p-values of about 0.1 or better due to the small sample size. Table 8 lists the significant differences when gender is the relevant variable. Put Table 8 about here Determining the difference between the means using a t-test is a pairwise procedure. The number of rounds played has three conditions: less than ten rounds, 10 to 12 rounds, and more than 12 rounds. The variables with a significant difference between the means of less than ten rounds and 10 to 12 rounds are shown in Table 9; Table 10 lists the variables with significant differences for the comparison between those playing less than ten rounds and those playing more than 12 rounds. TABLE 5 DIFFERENCES BETWEEN USA STUDENTS AND NON-USA STUDENTS Variable name “p” Value Your coursework completed prior to the simulation adequately prepared you to: make competitive decisions. ………………………………………………………………………………… “p” = 0.103 anticipate competitive reactions to my/(our) decisions. …………………………………….. “p” = 0.095 After "Playing" the simulation, I felt that that I: have the ability to make competitive decisions. …………………………………………………… “p” = 0.052 have the ability to create budgets. ………………………………………………….……………………. “p” = 0.019 have the ability to be innovative. …………………………………………………………………………. “p” = 0.100 Variable name “p” Value Your coursework completed prior to the simulation adequately prepared you to: set goals, determine actions to achieve the goals, and mobilize resources to execute the actions. …………………………..………………………..…… “p” = 0.022 After "Playing" the simulation, I felt that that I: work well in teams. ………………………………………………………………. “p” = 0.105 have the ability to understand the interactions between two or more decision variables. ……………………………………………………… “p” = 0.129 have the ability to do marginal analysis. ……………………………………….. “p” = 0.037 have the ability to anticipate competitive reactions to my/(our) firm's decisions. ………………………………………………………. “p” = 0.076 TABLE 6 DIFFERENCES BETWEEN THE TEAMS’ COMPETITIVE POSITION AT THE END OF THE SIMULATION. Page 93 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 Variable name “p” Value Your coursework completed prior to the simulation adequately prepared you to: work under uncertainty. ………………………………………………………. “p” = 0.115 After "Playing" the simulation, I felt that that I: had the ability to set goals, determine actions to achieve the goals, and mobilize resources to execute the actions. ……………………………….. “p” = 0.036 had the ability to differentiate important information from unimportant information. ……………………………………………………. “p” = 0.047 had the ability to work under uncertainty. ………………………………….. “p” = 0.036 had the ability to forecast outcomes, such as cash flows, units of ending inventory, unit demand, etc. ………………………………… “p” = 0.081 had the ability to understand the interactions among two or more decision variables. ……………………………………………….. “p” = 0.072 had the ability to assess risk. ………………………………………………….. “p” = 0.101 had the ability to consider possible competitors' decisions when making my/our firm's competitive decisions. …………………………. “p” = 0.011 had the ability to be innovative. ………………………………………………. “p” = 0.098 had the ability to be creative. …………………………………………………. “p” = 0.014 TABLE 7 DIFFERENCES BETWEEN SINGLE PLAYERS AND TEAM PLAYERS Variable name Men Women “p” value Your coursework completed prior to the simulation adequately prepared you to: make competitive decisions. ………………………. 5.63 5.13 0.013 forecast outcomes, such as cash flows, units of ending inventory, unit demand, etc. ……. 5.41 5 0.1 analyze reports and financial results. …………… 5.44 5 0.039 After playing" the simulation, I felt that that I have the ability to consider possible competitors' decisions when making my (our) firm's competitive decisions. …………………………….. 4.67 5.2 0.026 have the ability to set goals, determine actions to achieve the goals, and mobilize resources to execute the actions. ……………………………. 5.28 4.77 0.075 TABLE 8 DIFFERENCES BETWEEN MEN AND WOMEN Variable name Number of Rounds Played <10 10 to 12 “p” value Your coursework completed before the simulation adequately prepared you to: set goals, determine actions to achieve the goals, and mobilize resources to execute the actions. …….. 5.20 5.65 0.039 understand the interactions among two or more decision variables. ……………………………………. 5.07 5.53 0.058 After playing" the simulation, I felt that I had the ability to set goals, determine actions to achieve the goals, and mobilize resources to execute the actions…………………………………... 4.73 5.29 `0.077 TABLE 9 DIFFERENCES BETWEEN PARTICIPANTS “PLAYING: “LESS THAN 10 ROUNDS” AND THOSE “PLAYING 10 TO 12 ROUNDS” AND THEIR “P” VALUES Page 94 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 TABLE 10 DIFFERENCES BETWEEN PARTICIPANTS “PLAYING: “LESS THAN 10 ROUNDS” AND THOSE “PLAYING MORE THAN 12ROUNDS” AND THEIR “P” VALUES Variable name Number of Rounds Played <10 > 12 “p” value Your coursework completed before the simulation adequately prepared you to: work under uncertainty. …………………………….. 5.40 4.63 0.015 After playing" the simulation, I felt that I had the ability to analyze reports and financial results. ……………………………………… 5.53 4.88 0.029 had the ability to do marginal analysis. ……………. 5.00 4.38 0.050 TABLE 11 DIFFERENCES BETWEEN THE VARIABLE MEANS OF STUDENTS FROM THE USA AND INTERNATIONAL STUDENTS Variable name Means of Sudents from The USA Means of International Students “p” value Your coursework completed before the simulation adequately prepared you to: differentiate important information from unimportant information. ………………………..... 5.10 5.45 0.109 assess risk. .................................................................... 5.16 4.73 0.091 TABLE 12 THE DIFFERENCES BETWEEN THE TEAMS THAT FINISHED IN “FIRST PLACE” AND THOSE THAT FINISHED “IN THE MIDDLE OF THE PACK.” Variable Name Mean of students whose firms’ “placed First” Mean of students whose firms’ placed “in the middle Of the pack “p” value Your coursework completed before the simulation adequately prepared you to: set goals, determine actions to achieve the goals, and mobilize resources to execute the actions. …………………………….. 5.64 5.25 0.092 make competitive decisions. …………………….. 5.73 5.35 0.078 After playing" the simulation, I felt that I have the ability to do marginal analysis. ……… 5.36 4.50 0.002 have the ability to forecast outcomes. …………. 5.36 4.90 0.029 have the ability to analyze reports and financial results. ………………………………… 5.55 5.10 0.058 have the ability to understand the interactions among two or more decision variables. ………… 5.27 4.80 0.092 Page 95 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 Table 11 lists the variables with significant differences in the means between students from the USA and International students. Table 12 lists the variables with significant differences in the means between those finishing the simulation game in first place versus those finishing in the middle of the pack. Table 13 lists the variables with significant differences in the means between single participants and participants that were in teams CONCLUSIONS As this is only the results of a small-scale pilot study, we make no conclusions. However, we can claim that many more interactions exist among the different groups of students when they experience a business simulation than we previously thought. The authors expect to continue this study on a much larger and more representative sample of students and business programs within the coming year. TABLE 13 THE DIFFERENCES IN THE VARIABLE MEANS BETWEEN SINGLE PLAYERS AND THOSE PLAYING IN TEAMS Variable name Single player means Team player means “p” value Your coursework completed before the simulation adequately prepared you to: set goals, determine actions to achieve the goals, and mobilize resources to execute the action. ……….. 5.63 5.07 0.007 work under uncertainty. …………………………….... 4.05 5.33 0.079 After playing" the simulation, I felt that I have the ability to set goals, determine actions, to achieve the goals, and mobilize resources to execute the actions. ……………………………………. 5.15 4.60 0.045 have the ability to make competitive decisions. ……… 5.44 5.60 0.062 make differentiate important information from unimportant information. .……………………………… 5.33 5.10 0.088 have the ability to forecast outcomes, such as cash flows, units of ending inventory, unit demand, etc. …… 5.19 4.80 0.038 have the ability to understand the interactions among two or more decision variables. ……………….. 5.19 4.67 0.034 have the ability to analyze quality control measurements. …………………………………………. 5.22 4.60 0.066 have the ability to anticipate competitive reactions to my/(our) decisions. ………………………. 5.22 4.60 0.027 have the ability to consider possible competitors' decisions when making my/our firm's competitive decisions. ………………………..……….. 5.11 4.40 0.002 have the ability to be creative. ……………………….. 5.07 4.60 0.061 Page 96 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 Anderson, P. & Lawton, L. (2007). Simulation performance and its effectiveness as a PBL problem: A follow-up study. Developments in Business Simulation & Experiential Learning, 34. (Reprinted from Bernie Keys Library (11th ed.)) Anderson, P., Lawton, L., & Wellington, W. J. (2008). Goal orientation and simulation performance. Developments in Business Simulation & Experiential Learning, 35, 329-335. (Reprinted from Bernie Keys Library (11th ed.)) Anderson, P. & Lawton, L. (2009). Business simulations and cognitive learning: Developments, desires, and future directions. Simulation & Gaming, 40(2), 193-216. doi: 10.1177/1046878108321866 Armenakis, A., Feud H., & Holley, W. (1974). Correlates of satisfaction, learning, and success in business gaming. Simulations, Games and Experiential Learning Techniques, 1, 272-278. Chin, J., Dukes, R., & Gamson, W. (2009). Assessment in simulation and gaming: A review of the last 40 years. Simulation & Gaming, 40(4), 553-568. Dukes, R. & Seidner, C. (Eds.). (1978). Learning with simulations and games. Beverly Hills, CA: Sage Publications. Gosenpud, J. (1990). Evaluation of experiential learning. In J. W. Gentry (Ed.), Guide to business gaming and experiential learning (pp. 301-329). East Brunswick, NJ/London: Nichols/PB. Szot, J. (2013). The value and effectiveness of project management simulation as a capstone activity in an executive MBA educational program (Unpublished doctoral thesis). Skema Business School, Lille, France. Szot, J. (2016). Preplanning assignments to enhance learning when using the Sim4Projects project management simulation game as a capstone learning experience. Developments in Business Simulation and Experiential Learning, 44, 121-137. Taylor, J. & Walford, R. (1978). Learning and the simulation game. Beverly Hills, CA: Sage Publications. Teach, R. & Patel, V. (2007). Assessing participant learning in a business simulation. Developments in Business Simulation and Experiential Learning, 34, 76-84. Wolfe, J. & Willis, R. (2017). Who learns in a business game: a single player examination (Draft 3). Unpublished manuscript. REFERENCES