Forecasting Accuracy and Learning: The Key to Measuring Simulation Performance Developments in Business Simulation and Experiential Learning, Volume 33, 2006 FORECASTING ACCURACY AND LEARNING: THE KEY TO MEASURING SIMULATION PERFORMANCE Richard Teach Georgia Institute of Technology richard.teach@mgt.gatech.edu ABSTRACT This paper looks at forecasting errors made by student participants of the CAPSTONE simulation. CAPSTONE is a total enterprise simulation in which participants make individual product decisions as well as firm-wide management decisions. Over the eight rounds of the simulations, the student learned how to more accurately forecast outcomes. Each participant was essentially a “brand manager” for a single product and each student was held responsible for the contribution margin of their product. After the decisions for each round were made, each student was required to forecast the following four items: 1) the unit gross margin of their product; 2) the unit sale of their product; 3) their product’s market share; and 4) their product’s ending inventory levels in terms of the number of units on hand and the number of days of sales the inventory represented at the end of the round. The accuracy of these forecasts was then related to the student product’s contribution to overhead and profit. After the product level forecasts were made, the team acting as a committee of the whole forecast three firm- wide outcomes: 1) the cash–on-hand at the end of the period; 2) the return on sales (ROS) for the period and 3) the earnings per share (EPS) for the period. The study found a strong positive relationship between the product-level forecast accuracy and the product’s contribution margin and the firm-wide forecast accuracies and the firm’s profitability. A rather strange anomaly was found. If a firm went into a chapter 11 Condition (it needed an emergency loa), it became more profitable. Implications of these findings are discussed. THE CONCEPT The ability to adequately forecast the impact of changing key decision making variables must be learned before one can become a good practicing manager. Management by objectives would be impossible without a method of periodically assessing progress and using these assessments to forecast the ability to reach the final objectives. Managers constantly forecast on premises and assumptions they make about the future. Firm expansion decisions are based upon forecasts of increasing demand at profitable prices. The purchase of raw materials and component parts are based upon forecasts of production rates which, in turn, are based upon forecasts of future sales. The stockmarket constantly forecasts future expectations of firm sales and profits and, if a firm does not meet theses forecasts, the stock price almost always falls; it almost always increases, if the firm exceeds these forecasts or expectations. The choice of majors by university students are often impacted by forecasts of employment opportunities. To show this latter case, check enrollments in computer science after the dot-bomb situation in 2001. The steep price collapse of technology-based firms’ stock prices forecasted a drop in opportunity for students studying computer science. As a result, students changed majors and new students did not select computer science in the numbers that did in the 1990s. Note, the above discussion indicates that the ability to forecast events is a necessary but not a sufficient skill for managers. There is a long list of necessary managerial skills, and forecasting is but one of these many skills. The concept that forecasting is a key to management performance is not new. In 1987, Gregory Pickett and Roxanne Stell pointed out: “Forecasting is an accepted and necessary function performed to some degree by all businesses. Forecasts are used to help identify expected labor demand or wage rates, anticipated cash flow, future product sales, plant utilization, raw material usage, purchase requirements and general economic trends for use in strategic planning. Given the breadth of business activity affected by forecasted information, one might assume that a forecasting class would be a basic offering at most business colleges.” What is forecasting? Forecasting simply means “to calculate or predict (some future event or condition) usually the result of rational study and analysis of available pertinent data.”1 Forecasting itself is not technique dependent. It may involve very sophisticated statistical routines or econometric modeling or seat-of-the-pants estimation. This author has seen many types of highly successful forecasting techniques used in corporate environments. Newell Chiesl (1987) noted that forecasting methods vary in the degree of rigor and formality. And in Render and Stair’s (1982) text on forecasting, they have written, “In numerous firms the entire process is subjective, involving seat-of-the-pants methods, intuition and years of experience.” 48 mailto:richard.teach@mgt.gatech.edu Developments in Business Simulation and Experiential Learning, Volume 33, 2006 FORECASTING AND BUSINESS SIMULATIONS From a theoretical prospective, it would be extremely difficult to create a scenario in which forecasting was not an important component in the decision making process of a business simulation. Most total enterprise simulations require both a strategic planning process and decision making. Neither of these processes could proceed effectively unless the players forecast some form of competitive response from the strategic standpoint and the market-place response from the decision making perspective. For it to be otherwise, the participants would just be guessing or grasping at straws in the wind for direction. Numerous authors have written on the use of forecasting in business games. Over 15% of the 2125 papers in the 2005 edition of the Bernie Keys Library contained the word forecast. One of the more controversial articles was written by Richard Teach (1989) in which Teach suggested that business game performance could/should be measured by using forecasting accuracy and not enterprise profits. He concluded that if one could abandon using profits as the measure of success, the very nature of business simulations could change for the better. Business games could be designed that would make more realistic learning simulations that currently exist. Currently almost all business simulations start as with identical assets and equality among the firms and the marginal rates of return for each of the decision variables are equal across firms. The close relationship of forecasting accuracy to business simulation performance has been well documented. At the third ABSEL meeting, Jim Gentry and Edward Reutzel (1977) reported on an inventory control game written by Ronald Frazer. The game’s purpose was to give students an understanding of the complexity of the inventory control process. One of the key learning aspects of this game was for the students to “···devise a forecasting routine and incorporate it into the determination of the Economic Order Quantity (EOQ) and the reorder point” (p 224). In a paper relating forecasting abilities to business simulation performance, LaFollette and Belohlav (1981, p186) wrote: “The forecasting accuracy of each group (team) reflected the quality of the decisions that then determined the company’s performance. To put in another way, the accuracy of the forecasting of each group reflected that group’s effectiveness” Varanelli and Fazio (1981, p186) presented a paper at ABSEL in which they claimed that “···to forecast future results is crucial to being a successful game participant.” In an early study of indicators of success, Gosenpud, Miessing and Milton (1984) conducted a stepwise regression using return on investment (ROI) as the dependent variable with four independent variables: forecast accuracy, strategic stability, price strategy and formal by 106 students who had played THE EXECUTIVE GAME. The result showed the independent variable with the greatest Beta value was forecast accuracy which also had the lowest “p” value of less the 0.0005. The need for good forecasting when playing business games was also recognized by Newell Chiesl (1987, p30) when he wrote: “The value of the forecasting and planning technique is to provide a versimilitudinal (sic) experience for the students participating in a Business Simulation. The ultimate goal is to have students learn a planning technique. In order for the students to be successful in the computer game, they must be good record keepers, planners and forecasters.” Jerry Gosen and John Wasbush’s (2001) study of what is learned by students when they use a business simulation confirmed the strong link between forecasting accuracy and simulation performance (r=87 & p < 0.0005). Their finding that learning and forecasting accuracy were negatively related is not surprising, at least in this author’s opinion, as it indicated those who cannot forecast have a lot more to learn than students who already know how to forecast. In a 2002 paper, Washbush and Gosen had mixed results when comparing forecasting accuracy to simulation performance. In a Spring semester section of game players where the simulation went on for 13 periods, they found the relationship to have a coefficient of determination of 0.4142 with a “p” value of 0.0001, but in two sections taught in the Fall semester that ran for eight and nine periods, they reported lower coefficients of determination but unfortunately only reported the slope significance as N.S. instead of showing the actual “p” value. Thus the level of significance is unknown to the reader. John Washbush (2003, p 251) also reported findings that confirmed forecasting as highly related to simulation performance. He found, “Three correlations [between forecasting accuracy and performance] were statistically significant beyond the 0.01 level.” THE RELATIONSHIP BETWEEN FORECASTING ACCURACY AND FIRM PERFORMANCE HAS NOT BEEN ALWAYS FOUND Philip Anderson and Leigh Lawton (1990, p9), when studying the relationship between financial performance in business games and learning, found a 0.0 correlation between forecasting accuracy and financial performance when their students forecasted unit sales. However, they did find a relationship between forecasting accuracy and the team’s mean grade on a written analysis of the game 49 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 performance. Could this be an indication that forecasting accuracy was a measure of learning? As the forecast of unit contribution margin improves, the participants learn about the factors that drive unit variable costs. Forecasting of unit contribution margin drives home the importance of cost control and being aware of unit variable costs. As a result of the above background research, a study was designed in an attempt to measure forecasting errors as learning phenomena by individuals and the impact of forecasting accuracy upon performance from an individual as well as a team basis. The combination of unit sales forecast and unit contribution margin, when multiplied produces an estimate of the product’s total contribution margin for the period. If the firm manufacturers more than one product, then the sum of these values results in the firm’s overall contribution margin for the period. WHAT IS LEARNED BY HAVING FORECASTING AS A MAJOR COMPONENT OF A BUSINESS GAME If participants plot the gross margins by product by time period, it provides a methodology for anticipating the firms manufacturing costs and allows participants to check their assumptions about causes and affects of cash flows. The plots should also provide insight on the participants firm’s total profits and losses for each period. While accuracy in forecasting outcomes is very important in itself, there are other somewhat stealth learning outcomes that result from having participants produce forecasts of specific results. Below are seven specific forecasts and an analysis of what learning may take place as the accuracy of the forecasts improves. These are broken down into types: four forecasts about specific product measures and forecasts about firm level outcomes. FORECASTING EACH PRODUCT’S UNIT MARKET SHARE FOR EACH PERIOD Fortunately, simulation designers provide unambiguous information available to business simulation participants about industry level future demand. Accuracy of market share forecasts indicates that game participants are learning how to anticipate competitive responses, and the importance of competitive behavior in the market place. Initially, most business game respondents (at least this has been the experi- ence of this author)rely only on their own decisions when estimating their expected results and they often ignore the behavior of their competitors. This exercise of forecasting expected market shares by product by period focuses participants’ attention on all the players in the market place. FORECASTING UNIT SALES FOR EACH PRODUCT FOR EACH PERIOD As the forecasts improve, it indicates that the participant is understanding and learning what drives or causes unit sales, not only the decisions made within his/her firm, but also the competitive responses of all the firms in the marketplace and their affect on the participant firm’s unit sales. Accurate unit sales forecasts are necessary if rational manufacturing schedules are to be established and if adequate funding will be available. If the forecasts are inaccurate, excessive inventory or product “stock-outs” will occur, which adversely affects the firm’s performance and cask position. FORECASTING THE UNITS OF ENDING INVENTORY FOR EACH PRODUCT EACH PERIOD FORECASTING THE UNIT CONTRIBUTION MARGIN OF EACH PRODUCT THE FIRM SELLS FOR EACH PERIOD Forecasting ending inventory requires an understanding of expected unit sales, expected manufacturing levels and the prior period’s ending inventory. This should be an easy exercise because a unit sales forecast has already been generated, and the units of ending inventory are known. Thus, the simple equation of: Contribution margin is the dollar and cents that each unit of product sold contributes to the firm’s overhead costs and profits for the simulated period. The per-period unit contribution margin is very similar, though not identical, to the period’s average marginal cost. From a managerial point-of-view, the marginal cost is very important in managing a product and this “unit contribution margin” is an approximation to the marginal cost. There are a few occurrences where this approximation will fail, but in most cases it is a close estimate. While the exact marginal cost is rarely known in a simulation, (or in practice), the concept in the economics-of-the-firm (Micro) is overwhelmingly important. Manufacturing levelt = expected unit salest – ending inventoryt-1 + desired safety stockt (where the subscript t represents the particular simulation round) is in almost every operations and management strategy textbook. But, students constantly make substantial errors in this estimate. As the accuracy of the ending inventory forecast improves, the game participants may learn to apply 50 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 what has been taught to students in numerous business school courses. This learn by doing creates a substantial reduction in the teams’ cash management problems. FORECASTING THE RETURN ON SALES FOR THE FIRM FOR EACH PERIOD The forecast of unit contribution margin is an estimate of the dollars and cents return on a single unit of sales of a specific product. The estimated firm-wide ROS is a broad measure of the firm’s effectiveness. It measures the amount of profit that is generated by each dollar of sales (averaged across all products). If a team can accurately estimate or forecast the ROS for their firm, it should indicate they understand what drives this value. If they can accurately forecast ROS, they must be able to forecast the firm’s earnings. If they understand the cause and affect aspects of earnings and the drivers of sales, they should know how to increase the performance of the firm. The more accurate the forecast, the more they have learned about how the firm accrues profits. FORECASTING THE EARNINGS PER SHARE FOR THE FIRM FOR EACH PERIOD The Earnings per Share estimate is determined by taking the earnings estimate used in determining the forecast of ROS and dividing it by the number of shares of stock outstanding. Thus, ROS and EPS forecasts should be highly correlated. If one is much more accurate than the other, it would indicate that the participants have a misunderstanding that needs be corrected. FORECASTING THE ENDING CASH BALANCE FOR THE FIRM AT THE END OF EACH PERIOD Understanding cash flow is a critical skill in managing a firm. As such, being able to forecast the available cash at the end of each period of play in a business game indicates that the participant has learned the skill of cash management. The more accurate the forecast, the more the participant has learned. Many business simulations have an attribute that prevents bankruptcy. This feature exists in order that firms do not disappear from the competition. The logic of this characteristic is to prevent bankruptcy and keep all participants in the game for a limited number of rounds or periods. This design feature is necessary when the game is used in a situation where participant performance is evaluated by a function of the firm’s profits. This may have unintentional negative consequences. One of these negative consequences is that simulations often reduce the emphasis on cash management. This is a serious shortcoming because each time a firm needs an “emergency loan” to survive, the firm has actually gone into a n Chapter 11 bankruptcy, but this condition and its serious consequences is often unrecognized . This point is rarely pointed out by the simulation designers. A firm’s purpose is to maximize the shareholders value. Thus, when a firm is granted the “emergency loan” it has failed its stockholders and its creditors and the costs to its employees is never pointed out. What is bankruptcy? It is the inability for a firm to pay its bills in a timely manner. Participants, especially students, often this that bankruptcy is function of profitability, but it is not. It is a matter of cash management. Profitable firms can and do go Chapter 11 bankrupt if the run out of cash and cannot raise the needed cash in the short run. Thus, the accuracy of the participants forecasts are effective measures of the learning that takes place during the simulations’ run. THIS STUDY In most previous studies, there has been difficulty in measuring forecasting accuracy and simulation performance because forecasting is an individual’s skill and simulation performance is the result of team efforts. This paper reports on a study where each team participant was assigned a product to manage and each participant was competitively evaluated on her/his product’s contribution to overhead and profit as well as their team’s performance. THE GAME The data set used in this paper is from students in a B2B marketing course who played the game CAPSTONE (2004), which is a total enterprise simulation. The class had 18 teams playing the game under the “Footrace” scenario. In this “Footrace” scenario, each student team competed against five computer run firms. The computer team competitors were all set to be moderately competitive. The teams played two practice rounds and then eight rounds in competitive play. The game portion of each student’s final grade in the course was 20%. In CAPSTONE, each simulated company produces up to five products and each team had five participants. Each participant was assigned one product or brand that was his/her personal responsibility. Thus, each student would act as a Brand Manager for his/her assigned product and make all the decisions necessary to create the product, manufacture and market it to the specific target market. The students were told that their game grade would be based upon: 1) their product’s relative amount of total contribution margin compared to their 17 compatriots managing the identical brand in each of the other teams and 2) the relative market share of their product when compared to their 17 competitors. Each student’s performance was posted on the professor’s door after each round of play. 51 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 EXHIBIT 1 The product level forecasting form Firm number _______ Firm Name ______________________ for simulation year _______ The name of the person responsible for product Able? ______________________________ Unit Sales in the current year of product able? ____________ Units The expected unit sales next year for the product Able? ___________ Units The current unit gross margin of product Able $? _________.____ The expected unit gross margin next year for the product Able $? ___________.____ The current units of ending inventory of product Able? ___________ Units ________ Days The expected units of ending inventory of product Able (next year) _______ Units _____ Days The current market share of product Able? ______% (use at least one decimal place.) The expected market share of product Able for next year? ______% (use at least one decimal place.) Signature: __________________________________ EXHIBIT 2 The firm level forecasting form Firm results and projections What was the firm’s ROS (Return on sales) last year? _________% What do you expect your firm’s ROS will be next year? ___________% What was your firm’s EPS (Earnings per Share) last year? $______.____ What do you expect your firm’s EPS to be next year? $ ________.____ What is the current cask position of your firm? $ ___________ What do you project your firm’s total cash to be at the end of next year? $ _______________ Did your firm require an Emergency loan last year? [ ] Yes [ ] No Reviewed by the team Captain ____________________________________ Please print your name Signed by the team captain __________________________________________ Please sign you name 52 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 THE FORECASTING TASK Before each round was run, the teams handed in an “annual repot” of their analysis of the results of the prior round. At the end of this analysis was a set of forecasting forms in which each participant recorded the actual sales in units, the dollar unit gross margin of their product/brand, the ending inventory of product/brand and their market share of their product/brand in their product’s target market segment. Then, they forecast what they expected these values to be at the end of the next round of play. In addition, the each team recorded the rate of return on their companies’ sales (ROS), the firms’ earnings per share (EPS) and their cash balances. Each team was then required to forecast these same three values for the end of the following round of play. Exhibit 1 shows the form for the product Able (all products had the same required forecasts) and Exhibit 2 shows the form for the three firm level variables. Only the results of the product Able will be displayed. The results of all five products were very similar and the inclusion of the additional data would be redundant. Table 1 displays the average of the absolute vales of the error in forecasting of the four product-oriented forecasts by period. The absolute values of theses errors were computed to prevent any over-estimated forecasting errors from canceling out under-estimated forecasting errors. THE RESULTS The errors in forecasting at the individual product level were calculated and averaged for each of the eight periods of play of the game CAPSTONE. Table 1 displays the averaged forecasting errors for each of the four forecasts required for the product identified as Able. The first thing to notice is that in all four categories, the error terms went up in period 2. I believe this was a phenomenon of the two practice rounds that the teams completed before starting the eight competitive rounds. CAPSTON restarts the game with the same parameter and starting positions that exits in the practice rounds. As a rule, the students I have observed, when faced with a small set of practice rounds do not make drastic changes in their decisions when they begin their competitive game. Thus, they have experienced a similar outcome and their forecasts reflect this past experience. The average of the absolute values of the error terms peaked in the second period for three of the four forecasts. The average unit sales error term peaked in period three then dipped, increased and fell for the last three periods. The mean error term for forecasting unit contribution also peaked in the third period than then fell in each ensuing period with the exception of periods seven and eight. These last two increases in the average unit contribution forecasting error was the result of gigantic errors by only two of the 18 simulated firms. The errors in the market share forecasts bobbled around a little more than the other three error terms. In periods seven and eight, much of this increased error in estimating market share was due to two firms. Forecasting ending inventories also improved until period seven. Three firms were the culprits in forecasting their ending inventories in period seven, but in period eight, only one firm was off by a large proportion (see the footnotes in Table 1.). It was clear in the class that one firm no longer cared about the simulation and all members of this team were no longer willing to put in the time required to Arithmetic mean of absolute values of Simulation Year 2006 2007 Average Errors in Unit Sales forecasts 204 291 Average Errors in Unit Contribution $0.61 $0.62 Average Errors in Market Share Forecasts 4.50% 4.81% Average Error in Ending Inventory Forecasts7 142 430 Superscript 1 Firm 17 had a $5.71 er excluding firm 17 was Superscript 2 Firm 17 had a $3.57 fo excluding firm 17 was Superscript 3 Firms 17 had an over-f 3 had an over-forecast Superscript 4 Firms 17 had under for overestimated its Mark Superscript 5 Firms 17 had underest and Firm 3 by 643 unit Superscript 6 Firms 17 had underest Superscript 7 When calculating endin were excluded. TABLE 1 the forecasting errors for 18 teams Product ABLE 2008 2009 2010 2011 2012 2013 259 230 250 210 184 148 $0.75 $0.75 $0.63 $0.37 $0.891 $0.592 2.04% 1.65% 1.85% 1.54% 2.87%3 2.79% 4 250 204 184 175 3975 2276 ror in its forecast of unit gross margin. The average error $0.566, still above 2011 error, but below 2010’s error. recast error of unit contribution margin. The average error $0.396. orecast of Market Share by 8.1 percentage points and Firm of Market Share by 6.3% percentage points. ecast its Market Share by 4.7% and Firm 3 had et Share by 5.9%. imated its Ending Inventory by 2,140 units, Firm 12 by 1,287 s. These were the 3 greatest errors imated its Ending Inventory by 1145 units g inventory forecasts errors, all firms that had stock-outs 53 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 TABLE 2 The regression result in relating the dummy variables to the contribution to overhead and profit Non-standardized Coefficients Standardized “p” value Variable β Beta Constant 11,9509 <0.0005 D07 -1,350 -0.64 0.459 D08 2,497 0.113 0.184 D09 2,866 0.226 0.009 D10 9,585 0.445 <0.0005 D11 10,750 0.499 <0.0005 D12 7,028 0.326 <0.0005 D13 11,856 0.550 <0.0005 Model Summary R R2 Adjusted R2 Std Error 0.657 0.432 0.402 5,531 . ANOVA Sums of Squares df “p” value Regression 3.06E+09 7 <0.0005 Residual 4.01E+09 136 Total 7.06E+09 143 make effective decisions and more accurate forecasts. Their learning from the game came to a screeching halt. FORECASTING ERRORS AND PERFORMANCE The relationship between forecasting errors and product performance when product performance was measured by the total contribution to profit and overhead by product was the next issue. The hypothesis was that lower forecasting errors were directly related to the amount of contribution the product generated, period by period. To test this premise, a multiple regression was performed, using the product’s contribution as the dependent variable and using the absolute values terms of the four forecast errors as independent variables, Since CAPSTONE (as does almost all business games) has a growth factor built in as a set of exogenous parameters, a set of dummy variables were used to measure the affect of the growth parameters on the contribution term. There were seven dummy variables included to measure the market growth factors and were labeled D07, D08, D09, D10, D11, D12 and D13. Only seven dummy variables are needed to measures the eight growth parameters. As a first step, the set of dummy variables were regressed upon the total contri- bution and the residuals were saved. Table 2 displays the regression of the dummy variables on contribution. The residuals represented the contributions to overhead and profit, which are adjusted for growth due to the passage of time. Using the contributions adjusted for market growth as the dependent variable, the absolute values of the forecasting errors were used as independent variables in a stepwise regression. These results are shown in Table 3. For brevity, only the information of the last step of the regression analysis is shown. Readers should note that the errors in the ending inventory forecasts were excluded from this analysis because, when there was a “stock-out” condition, there were no readily available measures of the size of the forecasting error. Growth in the market place accounted for slightly over 40% of the variance in the contribution to overhead and profit. After excluding the growth factor, the absolute values of the three forecasting errors accounted for 40% of the variance of the growth adjusted contribution to overhead and profit. Everything else, including the stock-outs and the random error term, accounted for less than 60% of the variation of the contribution to overhead and profit. The above data analyzed each individual student’s accuracy in forecasting and related it to the performance of a single product, as measured by the product’s contribution to overhead and profit. The next section analyzes the corporate-wide forecasts of “ROS,” “EPS” and the “Cash on-hand at the end of each round” and relates these errors of these forecasts to the profitability of the firms. LOOKING AT CORPORATE-WIDE FORECASTING ERRORS AND ITS IMPACT ON PROFITABILITY Similar to the case in the product contribution to overhead and profit, the data needed to be adjusted to take out the effect of time on the firms’ profitability. Table 4 54 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 TABLE 3 Stepwise regression results using the residual values from the dummy variable regression as the dependent variable and the absolute values of the errors in the three forecasts as independent variables Model Unstandardized β Coefficients Standardized Beta “p” value Constant 4,426 < 0.0005 Absolute value of the error in forecasted unit gross margin - 2,910 - 0.676 < 0.0005 Absolute. value of the error in forecasted unit sales - 10.4 - 0.309 < 0.0005 Absolute. value of the error in forecasted market share - 402 - 0.269 0.024 Model Summary Model R R2 Adjusted R2 3 independent variables 0.657 0.432 0.402 ANOVA Model Sums of Squares df “p” value Regression 2.09E+09 3 < 0.0005 Residual 1.79E+09 113 Total 3.87E+09 143 TABLE 4 The regression result in relating the dummy variables to the profitability of the firms Variable Non-standardized β Coefficients Standardized Beta “p” value Constant 5,270 0.010 D07 1,329 0.038 0.644 D08 4,027 0.116 0.162 D09 10,030 0.288 0.0001 D10 13,158 0.378 <0.0005 D11 18,335 0.562 <0.0005 D12 15,552 0477 <0.0005 D13 23,650 0679 <0.0005 Model Summary R R2 Adjusted R2 Std Error 0.688 0.474 0.447 8,597 . ANOVA Sums of Squares df “p” value Regression 9.05E+09 7 <0.0005 Residual 1.01E+09 136 Total 1.91E+09 143 details the process of adjusting the firms’ profitability due to the passage of time. The residuals from this regression analysis, represents the firms’ profitability after taking the affect of time out of the data,. These residual value were then made the dependent variable of the second regression, which used the absolute values of the three forecasting errors, cash on hand at the end of each period of play, the ROS for each period of play, the EPS for each period of play and a dummy variable representing whether or not the firm went through a Chapter 11 bankrupts as independent variables.. These results are displayed in Table 5. The three forecasting error terms all had very significant, negative coefficients, therefore the larger the errors in forecasting, the lower the firms profits were. 55 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 TABLE 5 Stepwise regression result using the residual values from the dummy variable regression on per period profits as the dependent variable and the absolute values of the errors in the three forecasts as independent variables Unstandardized β Coefficients Standardized Beta “p” value Model β Constant 7,107 < 0.0005 Abs value of the error in the ROS forecast - 203,888 - 0.287 0.002 Abs. value of the error in cash forecast - 0.224 - 0.308 0.001 Chapter 11 condition 5,884 0.238 0.002 Abs. value of the error in forecasting EPS - 1,634 - 0.263 0.007 Model Summary Model R R2 Adjusted R2 4 independent variables 0.689 0.475 0.455 ANOVA Sums of Squares Df “p” value Regression 3.94E+09 4 <0.0005 Residual 3.60E+09 105 Total 7.03E+09 109 One would think that if a firm went through a “Chapter 11 bankruptcy its overall profitability should be negatively affected. But that was not the case in this CAPSTONE experience. It clearly indicates that when the firms in this one class of 18 firms and eight rounds has cash shortfalls, they were (on average) more profitable. RESULTS OF THE ANALYSIS The analysis shows that participants in CAPSTONE, which took part in this experience (I leave it up to the reader to generalize), performed much better when they learned how to forecast the outcomes of their decision processes. Also, as the game went on, they learned about the process of managing a simulated firm. Is this learning transferable to the practitioner world? That is beyond the capabilities of this data set, but this author can’t help but believe that it is. IMPLICATIONS This analysis has implications regarding what simulation participants learn and the links between learning and their firms’ performance. It also has implications for game design. If one measures game performance by forecasting abilities rather than profits, then games can be designed in more realistic way. We tend to fool ourselves into believing that our simulations are realistic because they produce income statements and balance sheets. However, what actual industry has every firm start with equal assets and exactly equal opportunities at the margin for each decision variable? Students should learn to leverage whatever their firm’s inherent advantages may be. Some firms have better R&D facilities than others. Some firms have access to lower costs of capital. Some firms are better marketers than others. Some can manufacture products with lower variable costs and /or lower overhead than others, while still others have superior design capabilities. One thing is certain, an industry where all firms have equal opportunities never exists. If one measures simulation performance by the ability to forecast outcomes, the nature of business games becomes much more like the real world. Also – end-play, where teams try to “beat” the game during the last period would disappear. There would be no reason to make drastic strategy or decision changes while making a last ditch attempt at winning. In fact, it would be best not to make drastic changes because the results of the drastic changes would be more difficult to forecast. The author encourages other teachers, researchers and users of business simulations to experiment with using a variety of measures other than the firm’s financial outcomes to evaluate student performance in a business game. although short run financial performance is used on Wall Street, it is not necessary to use the same indicators to measure student effectiveness. 56 Developments in Business Simulation and Experiential Learning, Volume 33, 2006 57 REFERENCES 1 Webster’s Collegiate Dictionary of American English, New York: Simon & Schuster, Inc. 1979, p445 _______ (2004) CAPSTONE, Business Simulation, Team Members Guide 2005, Management Simulations, Inc, Northfield, IL Anderson, Philip H. and Leigh Lawton (1990), “The relationship between financial performance and other measures of learning on a simulation exercise,” Developments in Simulation and Experiential Exercises, Vol.17 pp 6 - 10 Cheisl, Newell (1987), “The use of a simple forecasting technique during an interactive computerized business game,” , Developments in Business Simulation and Experiential Exercises, Vol. 14, pp 39 - 42 Gentry, James W. and Edward T. Reutzel (1977), “Teaching inventory control via computer interaction with increasing complexity,” Computer Simulation and Learning Theory, Vol. 3 1977 pp 219 - 228 Gosen, Jerry and John Washbush (2001), “An initial validity investigation of a test assessing total enterprise simulation learning,” Developments in Simulation and Experiential Exercises, Vol. 28 pp 92 - 95 Gosenpud, Jerry, Paul Miessing and Charles J. Milton (1984), “A research study on strategic decisions in a business simulation,” Developments in Simulation and Experiential Exercises, Vol.11 pp 161 -165 LaFollette, William R. and James A. Belohlav (1981), “Using simulation to study the effects of team compatibility,” Journal of Experiential Learning and Simulation Vol. 3, No. 4 pp181 - 190 Pickett, Gregory M. and Roxanne Stell (1987), “Teaching forecasting to the masses, Developments in Business Simulation and Experiential Exercises, Vol. 14, pp165 - 168 Render, Barry and Ralph. M. Stair, Jr. Quantitative Analysis for Management, Allyn and Bacon, 1982 Teach, Richard (1989),”Using forecasting accuracy as a measure of success in business simulations,” Developments in Business Simulation & Experiential Exercises, Vol. 16, pp. 103-107 Varanelli, Andrew, Jr. and Susan M. Fazio (1981), “A case study in the use of experiential learning to enhance student understanding of strategy evaluation and policy formulation,” Developments in Business Simulation & Experiential Exercises, Vol. 8, pp. 184 - 194 Washbush, John and Jerry Gosen (2002), “Total Enterprise simulation learning compared to traditional learning in the business policy course,” Developments in Business Simulation & Experiential Exercises, Vol. 29, pp. 281 - 286 Washbush, John B., (2003), “Simulation performance and forecast accuracy – is that all?” Developments in Business Simulation & Experiential Exercises, Vol. 30, pp. 250 – 253 Table of Contents Volume 33, 2006 Learning Assurance Using Business Simulations Applications To Executive Management Education Team Teaching In An Integrated Business Course Using Critical Problem Based Learning Factors In An Integrated Undergraduate Business Curriculum: A Business Course Success Personality Type And Strategic Planning Business Games As Strategic Management Laboratories The Relationship Between Students' Success On A Simulation Exercise And Their Perception Of Its Effectiveness As A PBL Problem Forecasting Accuracy And Learning: The Key To Measuring Simulation Performance The Business Strategy Game: A Performance Review Of The New Online Edition Using The Socratic Method And Bloom's Taxonomy Of The Cognitive Domain To Enhance Online Discussion, Critical Thinking, And Student Learning Using Negotiation Exercises To Promote Critical Thinking Skills Effective Leadership Experiences For Management Majors In A Futures Class The Role Of Learning Versus Performance Orientations When Reacting To Negative Outcomes In Simulation Games Is Pay Inversion Ethical? A Three-Part Exercise Simulations And Experiential Exercises - Do They Result In Learning? Have We Figured It Out Yet? Examining Program Management In Business Simulations: Student And Faculty Views Validating Business Simulations: Do Simulations Exhibit Natural Market Structures? Characterizing Business Games Used In Distance Education Utilizing Games In A Graduate Level Instructional Game Course Employment Interview Preparation: Assessing The Writing-To-Learn Approach Simulations - Bridging From Thwarted Innovation To Disruptive Technology Creating An Authentic Cultural Lens Using Case Dialogue Learning By Fire: Reflections Of A First Time Online Instructor An International Internship With A Service-Learning Focus Learner Participation In The Online Learning Experience: Help Or Hindrance? Any Given Sunday: Intervention In Pursuit Of Simulation Team Parity Beginning With The End: Creating An Experiential Exercise From Assessment Criteria Simulating Life Cycles: Life Span As The Measure Of Performance In Business Gaming Simulations It's Puzzling: Communications, Competition, And Cooperation Balanced Scorecard Implementation For Strategy Management: Variation Of Manager Opinion In Real And Simulated Companies Cases And Business Games: The Perfect Match! Three-Attribute Interrelationships For Industry-Level Demand Equations Using A Web-Based Module To Teach Information Literacy Decision Support System For Demand Forecasting In Business Games The Invalidity Of Profit=F(Market Share) PIMS Validation Of Marketing Games Online Market Test Laboratory With The MINSIM* Program The Gas Mileage Game - A Policy Simulation Delivered Cost And Differentiation Applied To Threshold 3rd Ed. The Effect Of Team-Leadership Modes On Team Performance: A Preliminary Study The Design And Use Of A Macroeconomics Simulation Using Maple Software: A Pilot Study The Instructor's Toolbox: A Meaning-Centered Framework For The Social Construction Of Experiential Learning Incorporating Strategic Product-Mix Decisions Into Simulation Games: Modeling The 'Profitable-Product Death Spiral' Group Composition And Groupthink In A Business Game A Direct Approach To Teaching Business Ethics A Decision Support System For Planning Sales, Production, And Plant Addition With Manager: A Computer Simulation Polish - American Entrepreneurial Business Cooperation Workshop Utilizing The Income/Outcome Simulation Student Leader Training Exercise Student Preference To Mode Of Learning In Hong Kong Experiential Learning For Technology-Based And Management Programme In Hong Kong: A China Study Tour A Price Game With Product Differentiation In The Classroom Discrete Event Modeling In A New Transportation Simulation Supply-Side Modeling In A Total Enterprise Simulation The Quality Game Towards A Massive Multiplayer Online Business Simulation Making The Connection: Improving Virtual Team Performance Through Behavioral Assessment Profiling And Behavioral Cues Individual Learning Producing A Learning Organization: 'Playing Dice With Polar Bears' Narratology and Ludology: Competing Paradigms or Complementary Theories in Simulation Framework For Evaluating Internet Research Using Children's Games To Illustrate Strategy Concepts: Is Less Better?