Using Accumulated Profits to Assess Performance in Simulations 183 Developments in Business Simulations and Experiential Learning, Volume 37, 2010 USING ACCUMULATED PROFITS TO ASSESS PERFORMANCE IN SIMULATIONS William D. Biggs Arcadia University biggs@arcadia.edu David J. Fritzsche The Business Policy Game fritzsch@eskimo.com ABSTRACT The issue of how to grade student performance in simulation environments has received considerable attention in the ABSEL literature. One line of argument, questions using accumulated profits as a basis for assessing performance. This article reviews and extends an earlier study and then discusses using accumulated profits to evaluate performance from a number of perspectives. INTRODUCTION For a number of years Teach (1987, 1990, and 1996 in Gentry, et al, 1996) has argued that profits, particularly cumulative profits, are not a valid measure of firm performance in simulation environments. A major aspect of his argument is that firms that get off to either a good or poor start, even if just due to good or bad luck, are destined to continue in the initial condition, due to the nature of simulation design. In a more recent work Teach and Patel (2007) provide research results addressing this issue. In the study reported here, we do a preliminary replication that extends the work of Teach and Patel (2007) in a number of ways. First, as Teach and Patel (2007) suggest in their call for further research, we test their hypothesis, using their method of calculations with a different simulation. Second, we extend the number of periods of play that are analyzed. Finally, we discuss using accumulated profits as the sole measure of performance in business simulations. THE TEACH AND PATEL (2007) STUDY Teach and Patel (2007) used the moderately complex total enterprise simulation, CAPSTONE® (Smith, 1997, 2001/2004) in undergraduate business to business marketing classes. They analyzed the data from 41 competitions with six teams per competition. Each competition was run for eight rounds. They ranked team performance on accumulated profits within each competition and then identified: (1) the winning firms by the round in which they took over the lead; and, (2) the last placed firms by the round in which they maintained last place. Teach and Patel (2007:79) also addressed the question, “If dominance occurred in cumulative profits, will ranking profits by round work any better?” They looked at the frequency of earning the highest per-round profits by noting the number of rounds firms finished in first place. They also show the number of times a firm which placed first also had a round in which that firm placed last. Since we provide the figures and table results from the Teach and Patel (2007) study with the current study results in order to make direct comparisons, we will not discuss their specific results at this point. Bernard and Souza (2009) have also conducted a similar study to Teach and Patel (2007). They found that dominance existed across a number of different simulations and participants. There are important differences, however, which negate direct comparison to Teach and Patel and the current study. First, and foremost they defined dominance differently. For them dominance existed if a team led more than 50% of the rounds whether the firm was in the lead at the end of the rounds or not. Second, they used stock price as the measure rather than accumulated profits. Third, they used manufacturing and retailing simulations rather than total enterprise simulations. Fourth, their competitions were run for 4, 7 or 8 rounds. THE CURRENT STUDY The current study used data from the 45th Annual International Collegiate Business Strategy Competition. This competition uses the Business Policy Game (Cotter and Fritzsche, 2005), a moderately complex total enterprise simulation. There were 19 teams of students from various universities who participated in the competition. They were divided into four worlds with six firms in World 1; six firms in World 2; four firms in World 3; and three firms in World 4. The competition was run for 20 rounds. Due to funding issues one of the firms in World 1 had to drop out of the competition at the end of the 11th round. The participants in Worlds 1 and 2 were undergraduate students and those in 184 Developments in Business Simulations and Experiential Learning, Volume 37, 2010 Worlds 3 and 4 were graduate students. It should be noted that a world in the current study corresponds to what Teach and Patel call a competition, so for some portions of the analysis we have only four data points to compare to their 41. RESULTS COMPARISONS In this section we present two figures and a table from the Teach and Patel (2007) study with data from the current study added to them. This approach facilitates making comparisons and aids in the discussion. Figure 1 presents by round of play when a firm first obtained, and then held first place in cumulative profits. While the current study had 20 rounds of play, we show the results at the end of eight rounds for direct comparison to Teach and Patel (2007), along with the results at the end of the 20 rounds of play. Figure 1 clearly supports their observation that 85% of the firms never relinquished the lead after round 3 of the 8 rounds. They comment that “This shows that dominance was a major factor in the business simulations analyzed.” The results of the current study, abet hampered by the small number of observations, are not quite as strong since this relationship held for only two of the four firms. This result suggests that the issue of dominance may be simulation specific. Of even greater interest, however, is the fact that the 20 round results show that three of the four firms obtained the first place position in much later rounds. Further, only one of the four firms that was dominate at the end of round 8 was still dominate at the end of round 20. Thus, an interesting question is whether the effects of dominance can be offset by more rounds of play? The longer timeframe may result in the poorer performing firm learning what needs to be done and thereby over take a Figure 1 Winning teams by the round in which they took over the lead 0 2 4 6 8 10 12 14 16 18 20 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Round Number N um be r o f F irm s Teach & Patel Number of Firms (41 Firms) End Round 8 Current Study Number of Firms (4 Firms) End Round 20 Current Study Number of Firms (4 Firms) End Round 8 185 Developments in Business Simulations and Experiential Learning, Volume 37, 2010 leading team. Conversely, it may be that the stronger performing team got there by luck, rather than knowledge and skill, and the longer timeframe means their lack of understanding catches up with them. A more intriguing possibility is that the poorer performing firm had a good strategy that needed time to become effective. The noise here for the current study is the one firm that dropped. This firm had attained first place at the end of round 6 and was still in first place at the end of round 11 when it had to drop out. At the end of round 8 including them means that the firm that held first place at the end of round 8 attained it at the end of round 6. If they are excluded the results are the same since the firm that would show up as finishing first would have attained first place at the end of round 6 and still would hold it at the end of round 8. There is a difference at the end of round 20, if they are included or excluded. If they are included the data is as is shown in Figure 1; if they are excluded, then the round 12 point for the first place leader shifts to round 6. Including them for this part of the analysis seems to make sense since we are looking at where all of the firms stood after 8 rounds of play. Figure 2 looks at firms that start out poorly and continue to perform poorly. Teach and Patel (2007) indicate that one firm was last in every period, three were last in all but round one, and 15 firms were last in all but the first two rounds. Thus, in nearly half of the competitions (19 of 41), firms that were in last place at the end of round three remained in last place through round eight. Again, the results of the current study are not as definitive, since only one of the four firms was in last place at the end of the first three rounds and continued in this state through round 8. However, the results through round 20 are not encouraging since by round 11, slightly better than half way through the rounds, the last place finisher on accumulated profits in each world was already established, and one of these firms was in last place in all 20 rounds. Thus, it may be hard to move out of last place. There is an important aside to mention here; the firm that was in last place for all 20 rounds was in World 4, which only had three firms. Is it possible that the more oligopolistic the industry, the harder it is to more out of last place? Table 1 presents data to address the question of whether ranking profits by round will work better than rankings based upon accumulated profits. Better in this context means that there will be less dominance so more firms will achieve first and/or last place positions. Thus, it looks at the Figure 2 Last placed firms by the first round in which they maintained last place 0 2 4 6 8 10 12 14 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Round Number N um be r o f F irm s Teach & Patel Number of Firms (41 Firms) End Round 8 Current Study Number of Firms (4 Firms) End Round 20 Current Study Number of Firms (4 Firms) End Round 8 186 Developments in Business Simulations and Experiential Learning, Volume 37, 2010 number of rounds in which a firm places first as well as whether those firms also had last place finishes. In looking at their results Teach and Patel 92007: 79) state: Obviously if a firm placed 1st in 8 rounds out of eight, they never placed last. There were eleven firms that placed 1st seven times, but of these eleven there was Table 1 Frequency of earning the highest per-round profits Teach and Patel Current Study # of Times in First Place Number of Firms Percent Firms in First Place This Number of Rounds1 # of Times these firms were in last place Percent Times in Last Place1 Number of Firms Percent Firms in First Place This Number of Rounds # of Times these firms were in last place Percent Times in Last Place 20 19 18 17 16 1 5.6% 1 1.3% 15 14 13 12 11 1 5.6% 1 1.3% 10 9 8 6 2.4% 0 0.0% 2 11.1% 2 2.5% 7 11 4.4% 1 0.3% 1 5.6% 3 3.8% 6 4 1.6% 1 0.3% 2 11.1% 4 5.0% 5 14 5.6% 1 0.3% 1 5.6% 1 1.3% 4 7 2.8% 1 0.3% 3 5 2.0% 5 1.6% 1 5.6% 0 0.0% 2 15 6.0% 5 1.6% 3 16.7% 21 26.3% 1 55 22.2% 61 18.9% 4 22.2% 31 38.8% Never First but were last 96 38.7% 247 76.7% 2 11.1% 16 20.0% Never first and never last 35 14.1% 0 0.0% 0 0.0% 0 0.0% Number of firms2,3 248 100.0% 18 100.0% Number of rounds2,3 322 100.0% 80 100.0% Notes: 1The percent columns were added for the Teach and Patel data. 2Teach and Patel had 41 competitions of 8 rounds each for a maximum number of rounds of 328; however, the data provided in the table only add to 322 rounds. The calculated percentages are based upon 322. They also indicate 6 firms per competition, which would be 246 firms; however, the numbers add to 248, which was used to calculate the percentages. 3The Current Study had 4 worlds of 20 rounds each for a maximum number of rounds of 80. We based the percentage calculations on the 18 firms that finished the competition. 187 Developments in Business Simulations and Experiential Learning, Volume 37, 2010 only one case in which the firm placed last during any round of play. There were 4 firms that placed 1st six times. Of the 8 opportunities in which they could have places last, only one firm placed last one time. Of the 14 firms that had the greatest profits 5 times, only one of these firms place last once out of 42 opportunities. Of the 7 firms that placed 1st in four rounds, again only one firm placed last one time out of 28 possibilities. It is striking that out of 328 opportunities, 96 firms never placed first, but these firms placed last in 247 rounds. Thirty-five firms never placed first or last in the 328 opportunities. In the current study 15 of the 18 firms that finished the competition (16 of the 19 who started) had at least one first place finish, and 15 of the 18 (16 of the 19 who started) had at least one last place finish, including all four of the firms that finished first based upon accumulated profits. The results show a significant contrast between the two studies in terms of the number of rounds teams who were never first but were last and the number of times these firms were in last place (76.7% verse 20.0%). Likewise, the studies differ on firms that were never first and never last (14.1% verse 0.0%). Thus, the results of the current study are not as definitive in terms of dominance as the Teach and Patel (2007) study when one looks at profits on a round by round basis. DISCUSSION Given that our results do not fully support the argument against using cumulative profits one might conclude that we disagree with Teach and Patel (2007) regarding using accumulated profits as the sole measure for assessing performance in simulations. In fact, however, we strongly agree that cumulative profits are not the best, or perhaps even an appropriate, measure to evaluate game performance, particularly if that measure is equated to student learning. We also strongly believe that multiple rather than single measures of performance, including non-financial measures, should be used in order to look at performance and student learning. We do disagree with their assertion that in the early years of simulation use “it was the accepted practice to use cumulative profits at the end-of-play as the student assessment tool for the game performance of a student’s grade in the course (Teach and Patel, 2007: 76).” Most researchers, even in the early years of simulations, argue for multiple measures, and in some cases, the use of non- financial measures in addition to financial measures, such as profit (Faria and Nulsen, 1974). Early on users argued for using multiple performance measures by combining a variety of items, such as sales in dollars, net income, ROS, ROA, EPS, stock price, etc. This approach was implemented by determining each firm’s ranking on each criterion and assigning some predetermined percent to that ranking. Parts A, B and C in Table 2 provide an example of this approach: Table 2 Example of Combined Criteria Score Part A: Absolute Score Sales in Dollars Net Income ROA1 Stock Price Firm 1 $1,500,000 $130,000 5.5% $2.50 Firm 2 $2,000,000 $140,000 6.0% $1.50 Firm 3 $1,000,000 -$50,000 7.0% $1.00 1ROA based on operating income Part B: Ranking Firm 1 2 2 3 1 Firm 2 1 1 2 2 Firm 3 3 3 1 3 Part C:Assigned Score Based on Rank (100 for 1; 80 for 2 and 60 for 3) Weighted Score Firm 1 80 80 60 100 80 Firm 2 100 100 80 80 90 Firm 3 60 60 100 60 70 Part D: Calculate Percent of Top Firm 1 75.0% 92.9% 78.6% 100.0% 86.6% Firm 2 100.0% 100.0% 85.7% 60.0% 86.4% Firm 3 50.0% -35.7% 100.0% 40.0% 38.6% As noted by Biggs (1978, 1976) a deficiency of the ranking approach is that it fails to recognize the magnitude of the differences among teams. This problem is illustrated in Part D of Table 2 by calculating each firm’s results for each criterion as a percentage of the highest performing firm for that criterion and equally weighting each criterion. In this relational approach it is clear that firms 1 and 2 are virtually identical in performance with firm 1 being the top performer whereas in the ranking approach firm 2 is the top performer, apparently by a substantial margin. In addition, the relational approach shows that firm 3 is a distant third, whereas, the ranking approach would suggest it is not that distant from firms 1 and 2. More recently, Bernard (2007, 2004) has looked at the relational and ranking approaches in the context of individual rather than team play. In recent years, a number of users have argued for using the Balanced Scorecard to assess student performance in simulations (see for example, Kallás and Sauaia, 2004 and Dickinson, 2003). In addition, to arguing for multiple performance measures to address student learning in simulations many users have argued for the use of non- simulation generated results such as reports, presentations, and boards of directors in order to assess student understanding and, therefore, learning (see for example, Anderson and Lawton, 1988). Independent of the results of the Teach and Patel (2007) study, an argument can be made against using cumulative profits on the basis of the nature of the calculations involved. The measure, cumulative profits, suffers from the problem of the base that is established after the first round of operations. This problem is illustrated in Table 3. 188 Developments in Business Simulations and Experiential Learning, Volume 37, 2010 Table 3 Absolute and Percent Change Calculations Period Absolute Net Income Period 0 Period 1 Period 2 Firm 1 $1,000,000 $ 1,100,000 $1,210,000 Firm 2 $1,000,000 $ 1,000,000 $1,310,000 Period Percent Change Net Income Firm 1 10.0% 10.0% Firm 2 0.0% 31.0% Cumulative Absolute Net Income Firm 1 $1,000,000 $ 2,100,000 $3,310,000 Firm 2 $1,000,000 $ 2,000,000 $3,310,000 Difference $ - $ 100,000 $ - In the illustration Firms 1 and 2 have the same net income in Period 0. In Period 1 Firm 1’s net income is 10% higher than it was in Period 0 while Firm 2’s is the same. On a cumulative basis Firm 1 has $100,000 (10%) more net income than Firm 2. If Firm 1 again attains a 10% increase in net income in Period 2, Firm 2 would have to obtain a 31% increase in Period 2 in order to have the same cumulative net income as Firm 1 at the end of Period 2. To the extent that both absolute and percent calculations are important, the mathematics of cumulative profits, almost by definition, may mean a firm that starts high gains a significant advantage, since the other firms are “chasing a moving target.” It appears it would more appropriate to look at period by period data rather than cumulative profits. It also appears to be strongly supportive of the view that multiple measures should be used to assess performance in business simulations. We would be remiss if we didn’t at least mention reasons for using accumulated profits. First, aren’t firms in the “real world” assessed to some extent on cumulative profits? The answer clearly is yes! We see quarterly reports which also include year to date profits, which suggests that cumulative profits are at least as important as “what have you done for me lately.” Second, isn’t it the case that firms in a particular industry frequently maintain their relative profit positions (i.e., firms in first place stay in first place and those in last place stay in last place)? For how many years has Avis said, “We’re number 2, so we try harder?” Do firms in industries tend to stay in the same relative positions over time, until a significant innovation or event occurs, as appears to be the case for firms in automobile, computer, and pharmaceutical industries? So if accumulated profits are appropriate for the “real world” why not for simulations? It seems to us that the reason is that our goal is different; we are not trying to merely assess performance but to facilitate and enhance learning. We want students to experiment with different approaches and techniques, which may result in lower profits, but still result in greater learning. One of the tenets of experiential learning is learning from ones mistakes. It can be argued that more learning can be achieved from mistakes than from successes. One could almost, but not quite, make an argument for rewarding the firm that has the lowest cumulative profit as it has made the most mistakes and possibly learned the most from the experience. In the “real world” we pay big bucks to people who are good at turn around strategies so why not reward this in simulations? This suggests we should look at change as well as absolute measures and that multiple measures should be used. The current study does not negate looking at other simulations to see if a measure such as cumulative profits generally results in dominance, since this may raise questions about the design of the simulation. Thus, we would encourage others to replicate these studies (with a larger n than in the current study). We would also like to see studies which look at the number of rounds of play and the number of directly competing firms to see if these variables make a difference. In doing any of these additional studies one must keep in mind the issue of the ranking approach. Thus, future studies might want to try to incorporate the relational approach to see if the differences really matter. REFERENCES Anderson, Philip H. and Lawton, Leigh (1988). “Assessing student performance on a business simulation exercise.” Developments in Business Simulation & Experiential Exercises, Vol. 15, 241-245. Reprinted in the Bernie Keys Library, 10th edition [Available from http://ABSEL.org] Bernard, Ricardo R. S. and de Souza, Moisés Pacheco (2009), “Dominance in online business games competitions.” Developments in Business Simulation and Experiential Learning, Vol. 36, 287-290. Reprinted in the Bernie Keys Library, 10th edition [Available from http://ABSEL.org] Bernard, Ricardo (2007). “Students’ perceptions on the individual manager performance.” Developments in Business Simulation and Experiential Learning, Vol. 34, 161-169. Reprinted in the Bernie Keys Library, 10th edition [Available from http://ABSEL.org] Bernard, Ricardo (2004). “Assessing individual performance in a total enterprise simulation.” Developments in Business Simulation and Experiential Learning, Vol. 31, 197-203. Reprinted in the Bernie Keys Library, 10th edition [Available from http://ABSEL.org] Biggs, William D. (1978). “A Comparison of ranking and relational grading procedures in a general management simulation.” Simulation & Gaming: An Interdisciplinary Journal of Theory, Practice and Research. Vol. 40, No. 5. Biggs, William D. (1976). “Relational approach to grading performance in business games.” Computer Simulation and Learning Theory, Vol. 3, 249-255. Reprinted in the 189 Developments in Business Simulations and Experiential Learning, Volume 37, 2010 Bernie Keys Library, 10th edition [Available from http://ABSEL.org] Cotter, Richard and Fritzsche, David (2005). BUSINESS POLICY GAME: AN INTERNATIONAL STRATEGY SIMULATION. http://www.eskimo.com/~fritzsch/game/bpg.html. Dickinson, John R. (2003). “The feasibility of the balanced scorecard for business games.” Developments in Business Simulation and Experiential Learning, Vol. 30, 90-98.Reprinted in the Bernie Keys Library, 10th edition [Available from http://ABSEL.org] Faria, Anthony J. and Nulsen, Jr., Ray O. (1974), “The use of internal and external assignments with a marketing simulation game.” Simulations, Games and Experiential Learning Techniques, Vol. 1, 54-60. Reprinted in the Bernie Keys Library, 10th edition [Available from http://ABSEL.org] Gentry, James W., Anderson, Phil, Bums, Alvin C., Cannon, Hugh, Faria, Anthony J, Frazer, Ron, Gosenpud, Jerry, Lawton, Leigh Nulsen, Ray, Teach, Richard, and Wellington, Bill (1996). “How do we measure the ‘learning’ in experiential learning and how do we best stimulate it?” Developments in Business Simulation & Experiential Exercises, Vol. 23, 197. Reprinted in the Bernie Keys Library, 10th edition [Available from http://ABSEL.org] Kallás, David and Sauaia, Antonio Carlos Aidar (2004). Implementation and impacts of the balanced scorecard: An experiment with business games. Developments in Business Simulation and Experiential Learning, Vol. 31, 242-248. Reprinted in the Bernie Keys Library, 10th edition [Available from http://ABSEL.org] Smith, Daniel (1997, 2001/2004). CAPSTONE®. http://www.capsim.com/. Teach, Richard D. (1987). “Profits: The false prophet.” Developments in Business Simulation & Experiential Exercises, Vol. 14, 205-207. Reprinted in the Bernie Keys Library, 10th edition [Available from http://ABSEL.org] Teach, Richard (1990). "Profits: The false profit in business games," Simulation and Gaming: An Interdisciplinary Journal of Theory, Practice and Research. Vol. 21, No. 1, 12-26. Teach, Richard and Patel, Vishal (2007). “Assessing participant learning in a business simulation.” Developments in Business Simulation and Experiential Learning, Vol. 34, 76-84. Reprinted in the Bernie Keys Library, 10th edition [Available from http://ABSEL.org] Table of Contents Volume 37, 2010 A Simulation in Organizational Behavior: Development and Beta Test Simulating Networks: An Experiment Pick Your Group Size: A Better Procedure to Resolve the Free-Rider Problem in a Business Simulation The Impact Of Playing A Marketing Simulation Game On Perceived Decision Making Ability Among Introductory Marketing Students If the Games Work, Why Aren't More Faculty Willing to Play? Modeling Cascading Demand: Accounting for the Effects of Captive Consumer Relationships Issues In Simulation Implementation: Lessons From a Freshman Seminar ENDURANCE: A game for current economic times Simulating Processes: An Application In Supply Chain Management Another Look At The Use Of Forecasting Accuracy On The Assessment Of Management Performance In Business Simulation Games The Effect Of Advertising On Demand In Business Simulations The Relationship Between Learning And Performance In A Total Enterprise Simulation: Revisited And New Data. Is the Gold /Pray Simulation Demand Model Valid and is It Really Robust? Mission Possible - Using Simulation Games For Management Training In A Transition Economy Business Game Modeling For The Negotiation Between General Contractor And Subcontractors The Relationship Between Goal Orientation And Simulation Performance With Attitude Change And Perceived Learning: A Follow-Up Study The Invalidity Of "Natural Market Structure" Pims Validation Of Marketing Games Examining A Beta Test An Experiment On Group Decision-Making Using A Business Game: An International Comparison Of Mba Students In Japan; China; And Russia A Platform for Business Games - From Game Playing to Game Making: The Case of Yokohama National University Checking Financial Balance of Target Brand Portfolio with the Strategic Market Plan Cash Flow Package About Simulations and Bloom's Learning Taxonomy Managing Complexity: Applying the Conscious-Competence Model to Experiential Learning Using Accumulated Profits to Assess Performance in Simulations And Now Let's Really Innovate: The Notion of Perpetual Online Courses Distance Learning: A Game Application Web Based Total Enterprise Simulation Learning Business Administration Using Simulation Software A Review of Experiential Exercises in Absel's Published Proceedings Moving Outside the Classroom: An Experiential Exercise Involving Video Clips Peer To Peer: A New Tool for Student Peer Evaluation Continuing To Assess Project Management: What's Been Done? What's It Means? Looking At New Twists Strategies for Promoting Knowledge Formation and Dissemination in Business Simulation and Experiential Learning Introducing Experiential Exercises To Inexperienced Faculty Members Conveying Concepts Through Cartoons: Preliminary Results From Software Engineers In An Industrial Organization Theory Class Utilizing Institutional Problems as an Experiential Learning Opportunity for Students Considerations in the Design of Vicarious Learning Environments Enhancing Critical Thinking Through Service-Learning As A Consultative Process Encouraging "Established" Faculty to Adopt a "New" Games and Simulations Pedagogy IMA Candy Company: An Experiential Exercise in Managerial Accounting Mustard Seeds as Means for Creative Problem Solving, Ethical Decision Making, Stakeholder Alliance, & Leader Development through Experiential Learning in Management Education Interactive Classroom Technology: Adding a Benefit to the Traditional Classroom Excelling in the Capstone Simulation: An Application of Spreadsheet Technology as a Decision Support Tool GEO: An Entrepreneurship-Oriented Computer-Assisted International Strategy Simulation ERP Simulation Game: A Distribution Game to Teach the Value of Integrated Systems