FORECASTING ACCURACY AND LEARNING: A KEY TO MEASURING BUSINESS GAME PERFORMANCE Developments in Business Simulation and Experiential Learning, Volume 34, 2007 57 FORECASTING ACCURACY AND LEARNING: A KEY TO MEASURING BUSINESS GAME PERFORMANCE Richard Teach Georgia Institute of Technology richard.teach@mgt.gatech.edu ABSTRACT This paper looks at forecasting errors made by student participants of the CAPSTONE simulation. CAPSTONE is a total enterprise simulation in which participants make individual product decisions as well as firm-wide management decisions. Over the eight rounds of the simulations, the student learned how to more accurately forecast outcomes. Each participant was essentially a “brand manager” for a single product and each student was held responsible for the contribution margin of their product. After the decisions for each round were made, each student was required to forecast the following four items: 1) the unit gross margin of their product; 2) the unit sale of their product; 3) their product’s market share; and 4) their product’s ending inventory levels in terms of the number of units on hand and the number of days of sales the inventory represented at the end of the round. The accuracy of these forecasts was then related to the student product’s contribution to overhead and profit. After the product level forecasts were made, the team acting as a committee of the whole forecast three firm-wide outcomes: 1) the cash–on-hand at the end of the period; 2) the return on sales (ROS) for the period and 3) the earnings per share (EPS) for the period. The study found a strong positive relationship between the product-level forecast accuracy and the product’s contribution margin and the firm-wide forecast accuracies and the firm’s profitability. A rather strange anomaly was found. If a firm went into a chapter 11 Condition (it needed an emergency loan), it became more profitable. Implications of these findings are discussed. THE CONCEPT The ability to adequately forecast the impact of changing key decision making variables must be learned before one can become a good practicing manager. Demand forecasts are a prerequisite for conducting productive sales and operations planning (Lapide 2005). Management by objectives would be impossible without a method of periodically assessing progress and using these assessments to forecast the ability to reach the final objectives. Managers constantly forecast on premises and assumptions they make about the future. Firm expansion decisions are based upon forecasts of increasing demand at profitable prices. The purchase of raw materials and component parts are based upon forecasts of production rates which, in turn, are based upon forecasts of future sales. The stockmarket constantly forecasts future expectations of firm sales and profits and, if a firm does not meet theses forecasts, the stock price almost always falls; it almost always increases, if the firm exceeds these forecasts or expectations. The choice of majors by university students are often impacted by forecasts of employment opportunities. To show this latter case, check enrollments in computer science after the dot-bomb situation in 2001. The steep price collapse of technology-based firms’ stock prices forecasted a drop in opportunity for students studying computer science. As a result, students changed majors and new students did not select computer science in the numbers that did in the 1990s. Note, the above discussion indicates that the ability to forecast events is a necessary but not a sufficient skill for managers. There is a long list of necessary managerial skills, and forecasting is but one of these many skills. The economic implications of learning by doing were outlined by Arrow (1962). Arrow was explaining forecasting manufacturing cost reductions through the utilization of learning curves as a forecasting process. This technique has learning as it central theme. It was learning itself that forecast cost reductions. The concept that forecasting is a key to a firm’s performance is not new. Gregory Pickett and Roxanne Stell (1987) pointed out: “Forecasting is an accepted and necessary function performed to some degree by all businesses. Forecasts are used to help identify expected labor demand or wage rates, anticipated cash flow, future product sales, plant utilization, raw material usage, purchase requirements and general economic trends for use in strategic planning. Given the breadth of business activity affected by forecasted information, one might assume that a forecasting class would be a basic offering at most business colleges.” (Quote page. 165) What is forecasting? Forecasting simply means “to calculate or predict (some future event or condition) usually the result of rational study and analysis of available pertinent data.”1 Forecasting itself is not technique dependent. It may involve very sophisticated statistical routines or econometric modeling or seat-of-the-pants mailto:richard.teach@mgt.gatech.edu Developments in Business Simulation and Experiential Learning, Volume 34, 2007 58 estimation. Rotenberg (1994), when discussing the forecasting of corporate financial performance, explained that there is a trend toward a greater reliance on more qualitative forecasting techniques. The results of a 1987 survey of Financial Executives reported that qualitative forecasts are creditable and informative to investors (Hoskin, Hughes and Ricks (1986). Skinner (1994) found an even greater proportion of non-quantitative forecasts. This author has seen many types of highly successful forecasting techniques used in corporate environments. Newell Chiesl (1987) noted that forecasting methods vary in the degree of rigor and formality. And in Render and Stair’s (1982) text on forecasting, they have written, “In numerous firms the entire process is subjective, involving seat-of-the- pants methods, intuition and years of experience,” (quote from page 18). Forecasting can help minimize inventory and obsolescence costs as well as reduce potential loss resulting from their stock-outs (Jain 2005/2006). Blue Cross and Blue Shield undertook a new strategy in 1992, one that was dedicated to learning. In that decree they cited, “Modeling and forecasting are becoming one of the most valuable tools for learning” (Ferguson 1992 p 27). The link between a reduction in forecasting errors and learning was made by Mansfield (1996) when he claimed that learning had taken place because forecasting errors had decreased. Since the late 1990s, there have been new measures of firm performance, namely, Economic Value Added (EVA) and it is claimed that this measure may be a better measure of firm performance than Earnings per share (EPS) (Ittner and Lacker 1998). However, EPS is still the preferred measure of firm performance. As a result, the new measure of EVA was investigated as a forecasting agent for EPS (Machuga, Pfeiffer and Veran 2002). These researchers found that Economic Value Added information could increase one’s ability to predict Earnings per Share and they also found that forecasting errors were related to EPS (The smaller the error the better the forecast.) Thus, increasing the ability to forecast would increase a firm’s financial performance. George Day (2002) wrote that organizations learn about their markets by market sensing and sense making. He added that “firms that have mastered these two processes gain an advantage by anticipating marketing opportunities ahead of their rivals and more accurately forecasting how the market will respond to their moves.” (p240) Thus forecasting is an important and a learning skill needed by successful firms. Not only is forecasting important, but it is becoming more important over time simply because of increased litigation concerns when firms publish their future expectations (Ciccone 2005). FORECASTING AND BUSINESS SIMULATIONS From a theoretical prospective, it would be extremely difficult to create a scenario in which forecasting was not an important component in the decision making process of a business simulation. Most total enterprise simulations require both strategic planning processes and decision making processes. Neither of these processes could proceed effectively unless the players forecast some form of competitive response from the strategic standpoint and the market-place response from the decision making perspective. For it to be otherwise, the participants would just be guessing or grasping at straws in the wind for direction. Capon and Palij (1994) reported that strategic forecasts were more accurate when greater levels of competition existed; and that superior forecasting performance was positively associated with superior firm performance. In most simulations, the periodic nature of receiving data feedback is such that only a few data points are available for forecasting purposes. In general the degrees of freedom are such that the number of independent variables used for forecasting purposes in very limited. As a result most forecasting in business games is the result of using simplistic and not complex methodologies. But, a study reported by Homes, et al. (2005) suggests simple models may be more useful in the learning process. Their research found that the complex forecasting strategies, as compared to simpler techniques slowed down the learning process. Numerous authors have written on the use of forecasting in business games. One of the more controversial articles was written by Richard Teach (1993) in which he suggested that business game performance could/should be measured by using forecasting accuracy and not enterprise profits. He concluded that if one could abandon using profits as the measure of success, the very nature of business simulations could change for the better. Business games could be designed that would make more realistic learning simulations that currently exist. Currently almost all business simulations start with identical assets and equality among the firms and the marginal rates of return for each of the decision variables are equal across firms. Jim Gentry and Edward Reutzel (1977) reported on an inventory control game written by Ronald Frazer. Thy reported that the purpose of Frazer’s game was to provide students with an understanding of the complexity of the inventory control process. One of the key learning aspects of this game was for the students to “···devise a forecasting routine and incorporate it into the determination of the Economic Order Quantity (EOQ) and the reorder point (Quote from page 224). Here again, accurate forecasting leads to grater firm performance and learning to forecast is a major attribute of playing a simulation.. In a paper relating forecasting abilities to business simulation performance, LaFollette and Belohlav (1981, p. 186) wrote: “The forecasting accuracy of each group (team) reflected the quality of the decisions that then determined the company’s performance. To put in Developments in Business Simulation and Experiential Learning, Volume 34, 2007 59 another way, the accuracy of the forecasting of each group reflected that group’s effectiveness” Varanelli and Fazio’s paper (1981, p186) claimed that “···to forecast future results is crucial to being a successful game participant.” In an early study of indicators of success, Gosenpud, Miessing and Milton (1984) conducted a stepwise regression using return on investment (ROI) as the dependent variable with four independent variables. These variables were: forecast accuracy, strategic stability, price strategy and formal planning. The data for this study were supplied by 106 students who had played THE EXECUTIVE GAME. The result showed the independent variable with the greatest Beta value was forecast accuracy which also had the lowest “p” value of less the 0.0005. The need for good forecasting when playing business games was also recognized by Newell Chiesl (1987 when he wrote: “The value of the forecasting and planning technique is to provide a versimilitudinal (sic) experience for the students participating in a Business Simulation. The ultimate goal is to have students learn a planning technique. In order for the students to be successful in the computer game, they must be good record keepers, planners and forecasters,” (quote from page 30). A study conducted at Columbia University on MBA students playing a business game (Glazer, Steckel, and Winer.1989); found that the degree of firm rationality in the forecasting process and the level of forecasting accuracy were directly related to performance. Jerry Gosen and John Wasbush’s (2001) study of what is learned by students when they use a business simulation confirmed the strong link between forecasting accuracy and simulation performance (r = 0.87 & p < 0.0005). Their finding that learning and forecasting accuracy were negatively related is not surprising, at least in this author’s opinion, as it indicated those who cannot forecast have a lot more to learn than students who already know how to forecast. In a 2002 paper, Washbush and Gosen had mixed results when comparing forecasting accuracy to simulation performance. In a Spring semester section of game players where the simulation went on for 13 periods, they found the relationship to have a coefficient of determination of 0.4142 with a “p” value of 0.0001, but in two sections taught in the Fall semester that ran for eight and nine periods, they reported lower coefficients of determination but unfortunately only reported the slope significance as N.S. instead of showing the actual “p” value. Thus, the level of significance is unknown to the reader. John Washbush (2003, p 251) also reported findings that confirmed forecasting as highly related to simulation performance. He found, “Three correlations [between forecasting accuracy and performance] were statistically significant beyond the 0.01 level.” THE RELATIONSHIP BETWEEN FORECASTING ACCURACY AND FIRM PERFORMANCE IN BUSINESS GAMES HAS NOT BEEN ALWAYS FOUND Philip Anderson and Leigh Lawton (1992, p334) when studying the relationship between financial performance in business games and learning, found a 0.0 coefficient of determination between forecasting accuracy and financial performance when their students forecasted unit sales. Note that it is extremely difficult to find a zero R2 between any two sets of observations. However, they did find a relationship between forecasting accuracy and the team’s mean grade on a written analysis of the game performance. Could this be an indication that forecasting accuracy was a measure of learning? As a result of the above background research, a study was designed in an attempt to measure forecasting errors as learning phenomena by individuals and the impact of forecasting accuracy upon performance from an individual as well as a team basis. LEARNING BY FORECASTING In an article on agent based manufacturing systems, Shen, Maturana and Norrie (2000) divided the learning process into “learning from history,” the learning from cases and reasoning and “learning from the future,” which was learning from system forecasting simulations. While accuracy in forecasting outcomes is very important in itself; there are other somewhat stealth learning outcomes that result from having participants produce forecasts of specific results. Below are seven specific forecasts and an analysis of what learning may take place as the accuracy of the forecasts improves. These are broken down into types: four forecasts about specific product measures and three forecasts about firm level outcomes. FORECASTING UNIT SALES FOR EACH PRODUCT FOR EACH PERIOD As the forecasts improve, it indicates that the participant is understanding and learning what drives or causes unit sales, not only the decisions made within his/her firm, but also the competitive responses of all the firms in the marketplace and their affect on the participant firm’s unit sales. Accurate unit sales forecasts are necessary if rational manufacturing schedules are to be established and if adequate funding will be available, since sales are the cash generator for most established firms. If the forecasts are inaccurate, excessive inventory or product “stock-outs” will occur, which adversely affects the firms’ performance and cash positions. Developments in Business Simulation and Experiential Learning, Volume 34, 2007 60 FORECASTING THE UNIT CONTRIBUTION MARGIN OF EACH PRODUCT THE FIRM SELLS FOR EACH PERIOD Contribution margin is the dollar and cents that each unit of product sold contributes to the firm’s overhead costs and profits for the simulated period. The per-period unit contribution margin is very similar, though not identical, to the period’s dollar sales less the average marginal cost times the unit sales. While the exact marginal cost is rarely known in a simulation, (or in practice) the concept in the economics-of-the-firm (Micro) is overwhelmingly important. As the forecast of unit contribution margin improves, the participants learn the factors that drive unit variable costs. Forecasting of unit contribution margin drives home the importance of cost control and being aware of unit variable costs. The combination of unit sales forecast and unit contribution margin, when multiplied produces an estimate of the product’s total contribution margin for the period. If the firm manufactures more than one product, then the sum of these product contributions results in the firm’s overall contribution margin for the period. If participants plot the gross margins by product by time period, it provides a methodology for anticipating the firms manufacturing costs and allows the players to check their assumptions about causes and affects of its cash flows. The plots should also provide insight on the participant firm’s total profits and losses for each period. FORECASTING EACH PRODUCT’S UNIT MARKET SHARE FOR EACH PERIOD Fortunately, simulation designers provide unambiguous information available to business simulation participants about industry level future demand. Accuracy of market share forecasts indicates that game participants are learning how to anticipate competitive responses, and the importance of competitive behavior in the market place. Initially, most business game respondents (at least this has been the experience of this author) rely only on their own decisions when estimating their expected results and they often ignore the behavior of their competitors. This exercise of forecasting expected market shares by product by period focuses participants’ attention on all the players in the market place. FORECASTING THE UNITS OF ENDING INVENTORY FOR EACH PRODUCT EACH PERIOD Forecasting ending inventory requires an understanding of expected unit sales, expected manufacturing levels and the prior period’s ending inventory. This should be an easy exercise because a unit sales forecast has already been generated, and the units of ending inventory are known. Thus, the simple equation of: Manufacturing levelt = expected unit salest – ending inventoryt-1 + desired safety stockt (Where the subscript t represents the particular simulation round or time period) is in almost every operations and management strategy textbook. But, students constantly make substantial errors in this estimate. As the accuracy of the ending inventory forecast improves, the game participants may learn to apply what has been taught to students in numerous business school courses. This learn by doing creates a substantial reduction in the teams’ cash management problems. The amount of safety stock needed is a function of forecasting accuracy and the risk of running out of inventory (having stock-outs) balanced by the cost of carrying inventory. FORECASTING THE RETURN ON SALES FOR THE FIRM FOR EACH PERIOD The forecast of unit contribution margin is an estimate of the dollars and cents return on a single unit of sales for a specific product. The estimated firm-wide ROS is a broad measure of the firm’s effectiveness. It measures the amount of profit that is generated by each dollar of sales (averaged across all products). If a team can accurately estimate or forecast the ROS for their firm, it should indicate that they understand what drives this value. If they can accurately forecast ROS, they must be able to forecast the firm’s earnings. If they understand the cause and affect aspects of earnings and the drivers of sales, they should know how to increase the performance of the firm. The more accurate the forecast, the more they have learned about how the firm accrues profits. FORECASTING THE EARNINGS PER SHARE FOR THE FIRM FOR EACH PERIOD The earnings per share estimate is determined by taking the earnings estimate used in determining the forecast of ROS and dividing it by the number of shares of stock outstanding. Thus, ROS and EPS forecasts should be highly correlated. If one is much more accurate than the other, it would indicate that the participants have a misunderstanding that needs be corrected. FORECASTING THE ENDING CASH BALANCE FOR THE FIRM AT THE END OF EACH PERIOD Understanding cash flow is a critical skill in managing a firm. As such, being able to forecast the available cash at the end of each period of play in a business game indicates that the participant has learned the skill of cash management. The more accurate the forecast, the more the participant has learned. Many business simulations have an attribute that prevents bankruptcy. This feature exists in order that firms do not disappear from the competition. The logic of this characteristic is to prevent bankruptcy and keep all participants in the game for a limited number of rounds or periods. This design feature is necessary when the game is Developments in Business Simulation and Experiential Learning, Volume 34, 2007 61 used in a situation where participant performance is evaluated by a function of the firm’s profits. This may have unintentional negative consequences. One of these negative consequences is that simulations often reduce the emphasis on cash management. This is a serious shortcoming because each time a firm needs an “emergency loan” to survive, the firm has actually gone into a Chapter 11 bankruptcy, but this condition and its serious consequences is often unrecognized. This point is rarely pointed out by the simulation designers. A firm’s purpose is to maximize the shareholders value. Thus, when a firm is granted the “emergency loan” it has failed its stockholders and its creditors and the costs to its employees are almost never pointed out. What is bankruptcy? It is the inability for a firm to pay its bills in a timely manner. Participants in business games, especially undergraduate students, often think that bankruptcy is function of profitability, but it is not. It is a matter of cash management. Profitable firms can and do go through Chapter 11 reorganizations and sometimes Chapter 7 proceedings if they have a poor chance of reaching a positive cash flow. Thus, the accuracy of participant’s forecasts is an effective measure of the learning that takes place during a simulation experience. This may well be the highest greatest use of business simulations. THIS STUDY In most previous studies, there has been difficulty in measuring forecasting accuracy and simulation performance because forecasting is an individual’s skill and simulation performance is the result of team efforts. This paper reports on a study where each team participant was assigned a product to manage and each participant was competitively evaluated on her/his product’s contribution to overhead and profit as well as their team’s performance. THE GAME The data set used in this paper is from students in a B2B marketing course that played the game CAPSTONE (2004), which is a total enterprise simulation. The teams played two practice rounds and then eight rounds in competitive play. The game portion of each student’s final grade in the course was 40%. In CAPSTONE, each simulated company produces up to five products and each team had five participants. Each participant was assigned one product or brand that was his/her personal responsibility. Thus, each student would act as a Brand Manager for his/her assigned product and make all the decisions necessary to create the product, manufacture and market it to the specific target market. The students were told that their game grade would be based upon: 1) their product’s relative amount of total contribution margin compared to their compatriots managing the identical brand in each of the other teams and 2) the relative market share of their product when compared to their competitors. Each student’s performance was posted on the professor’s door after each round of play. THE FORECASTING TASK Before each round was run, the teams handed in an “annual report” of their analysis of the results of the prior round. At the end of this analysis was a set of forecasting forms in which each participant recorded the actual sales in units, the dollar unit gross margin of their product/brand, the ending inventory of product/brand and their market share of their product/brand in their product’s target market segment. Then, they forecast what they expected these values to be at the end of the next round of play. In addition, the each team EXHIBIT 1 The product level forecasting form Firm number _______ Firm Name ______________________ for simulation year _______ The name of the person responsible for product Able. ______________________________ Unit Sales in the current year of product able ____________ Units The expected unit sales next year for the product Able ___________ Units The current unit gross margin of product Able $ _________.____ The expected unit gross margin next year for the product Able $ ___________.____ The current units of ending inventory of product Able ___________ Units ________ Days The expected units of ending inventory of product Able (next year) _______ Units _____ Days The current market share of product Able ______% (use at least one decimal place.) The expected market share of product Able for next year ______% (use at least one decimal place.) Signature: __________________________________ Developments in Business Simulation and Experiential Learning, Volume 34, 2007 62 recorded the rate of return on their companies’ sales (ROS), the firms’ earnings per share (EPS) and their cash balances. Each team was then required to forecast these same three values for the end of the following round of play. In industry, forecasts are updated many times in a year, (Landry et al 2005). Many simulations have either a one quarter or one year accounting and decision cycle. Thus, in a game using one year decision cycles, does not allow for any updates to be made to annual forecasts; a situation that never occurs in an actual practicing firm. Exhibit 1 shows the form for the product Able (all products had the same required forecasts. Exhibit 2 shows the form for the three firm level variables. THE RESULTS Only the results of the product Able will be displayed. The results of all five products were very similar and the inclusion of the additional data would be redundant. EXHIBIT 2 The firm level forecasting form Firm results and projections What was the firm’s ROS (Return on sales) last year? _________% What do you expect your firm’s ROS will be next year? ___________% What was your firm’s EPS (Earnings per Share) last year? $______.____ What do you expect your firm’s EPS to be next year? $ ________.____ What is the current cask position of your firm? $ ___________ What do you project your firm’s total cash to be at the end of next year? $ _______________ Did your firm require an Emergency loan last year? [ ] Yes [ ] No Reviewed by the team Captain ____________________________________ Please print your name Signed by the team captain __________________________________________ Please sign you name TABLE 1 Arithmetic mean of absolute values of the forecasting errors for 18 teams Product ABLE Simulation Year 2006 2007 2008 2009 2010 2011 2012 2013 Average Errors in Unit Sales forecasts 204 291 259 230 250 210 184 148 Average Errors in Unit Contribution $0.61 $0.62 $0.75 $0.75 $0.63 $0.37 $0.891 $0.592 Average Errors in Market Share Forecasts 4.50% 4.81% 2.04% 1.65% 1.85% 1.54% 2.87%3 2.79% 4 Average Error in Ending Inventory Forecasts7 142 430 250 204 184 175 3975 2276 Superscript 1 Firm 17 had a $5.71 error in its forecast of unit gross margin. The average error excluding firm 17 was $0.566, still above 2011 error, but below 2010’s error. Superscript 2 Firm 17 had a $3.57 forecast error of unit contribution margin. The average error excluding firm 17 was $0.396. Superscript 3 Firms 17 had an over-forecast of Market Share by 8.1 percentage points and Firm 3 had an over-forecast of Market Share by 6.3% percentage points. Superscript 4 Firms 17 had under forecast its Market Share by 4.7% and Firm 3 had overestimated its Market Share by 5.9%. Superscript 5 Firms 17 had underestimated its Ending Inventory by 2,140 units, Firm 12 by 1,287 and Firm 3 by 643 units. These were the 3 greatest errors Superscript 6 Firms 17 had underestimated its Ending Inventory by 1145 units Superscript 7 When calculating ending inventory forecasts errors, all firms that had stock-outs were excluded. Developments in Business Simulation and Experiential Learning, Volume 34, 2007 63 Table 1 displays the average of the absolute vales of the error in forecasting of the four product-oriented forecasts by period. The absolute values of theses errors were computed to prevent any over-estimated forecasting errors from canceling out under-estimated forecasting errors. The errors in forecasting at the individual product level were calculated and averaged for each of the eight periods of play of the game CAPSTONE. Table 1 displays the averaged forecasting errors for each of the four forecasts required for the product identified as Able. The first thing to notice is that in all four categories, the error terms went up in period 2. I believe this was a phenomenon of the two practice rounds that the teams completed before starting the eight competitive rounds. CAPSTON restarts the game with the same parameter and starting positions that exits in the practice rounds. As a rule, the students I have observed, when faced with a small set of practice rounds do not make drastic changes in their decisions when they begin their competitive game. Thus, they have experienced a similar outcome and their forecasts reflect this past experience. The average of the absolute values of the error terms peaked in the second period for three of the four forecasts. The average unit sales error term peaked in period three then dipped, increased and then fell for the last three periods. The first thing to notice is that in all four categories, the error terms went up in period 2. I believe this was a phenomenon of the two practice rounds that the teams completed before starting the eight competitive rounds. CAPSTON restarts the game with the same parameter and starting positions that exits in the practice rounds. As a rule, the students I have observed, when faced with a small set of practice rounds do not make drastic changes in their decisions when they begin their competitive game. Thus, they have experienced a similar outcome and their forecasts reflect this past experience. The average of the absolute values of the error terms peaked in the second period for three of the four forecasts. The average unit sales error term peaked in period three then dipped, then increased and finely fell for the last three periods. The mean error term for forecasting unit contribution also peaked in the third period than then fell in each ensuing period with the exception of periods seven and eight. These last two increases in the average unit contribution forecasting error was the result of gigantic errors by only two of the 18 simulated firms. The errors in the market share forecasts fluctuated a little more than the other three error terms. In periods seven and eight, much of this increased error in estimating market share was due to two firms. Forecasting ending inventories also improved until period seven. Three firms had very large forecasting errors in this area, but in period eight, only one firm was off by a large proportion (see the footnotes in Table 1.). It was clear in the class that one firm no longer cared about the simulation and all members of this team were no longer willing to put in the time required to make effective decisions and more accurate forecasts. Their learning from the game came to a screeching halt. FORECASTING ERRORS AND PERFORMANCE The relationship between forecasting errors and product performance when product performance was measured by the total contribution to profit and overhead by product was TABLE 2 Stepwise regression results using the contribution to overhead and profit as the dependent variable and the absolute values of the error terns in the three forecasts as independent variables Model Unstandardized β Coefficients Standardized Beta “p” value Constant 4,426 < 0.0005 Absolute value of the error in forecasted unit gross margin - 2,910 - 0.676 < 0.0005 Absolute. value of the error in forecasted unit sales - 10.4 - 0.309 < 0.0005 Absolute. value of the error in forecasted market share - 402 - 0.269 0.024 Model Summary Model R R2 Adjusted R2 3 independent variables 0.657 0.432 0.402 ANOVA Model Sums of Squares df “p” value Regression 2.09E+09 3 < 0.0005 Residual 1.79E+09 113 Total 3.87E+09 143 Developments in Business Simulation and Experiential Learning, Volume 34, 2007 64 the next issue. The proposition was that lower forecasting errors were directly related to the amount of contribution the product generated, period by period. To test this premise, a multiple regression was performed, using the product’s contribution as the dependent variable and using the absolute values terms of the four forecast errors as independent variables. These results are shown in Table 2. For brevity, only the information of the last step of the regression analysis is shown. Readers should note that the errors in the ending inventory forecasts were excluded from this analysis because, when there was a “stock-out” condition, there were no readily available measures of the size of the forecasting error. The absolute values of the three forecasting errors accounted for 40% of the variance of the contribution to overhead and profit. Growth in the market place accounted for slightly over 40% of the variance in the contribution to overhead and profit. Everything else, including the stock- outs and the decision and strategy errors made by the firms, accounted for less than 20% of the variation of the contribution to overhead and profit. The above data analyzed each individual student’s accuracy in forecasting and related it to the performance of a single product, as measured by the product’s contribution to overhead and profit. The next section analyzes the corporate-wide forecasts of “ROS,” “EPS” and the “Cash on-hand at the end of each round” and relates these errors of these forecasts to the profitability of the firms. LOOKING AT CORPORATE-WIDE FORECASTING ERRORS AND ITS IMPACT ON PROFITABILITY Firm profitability was then made the dependent variable of the second regression, which used the absolute values of the three forecasting errors, cash on hand at the end of each period of play, the ROS for each period of play, the EPS for each period of play and a dummy variable representing whether or not the firm went through a Chapter 11 bankrupts (ran out of cash) as independent variables.. These results are displayed in Table 3. The three forecasting error terms all had very significant, negative coefficients, therefore the larger the errors in forecasting, the lower the firms profits were. One would think that if a firm went through a “Chapter 11” bankruptcy its overall profitability should be negatively affected. But that was not the case in this CAPSTONE experience. It clearly indicated that when the firms in this one class of 18 firms simulated for eight rounds had cash shortfalls, they were (on average) more profitable. RESULTS OF THE ANALYSIS The analysis shows that participants in CAPSTONE, which took part in this experience (I leave it up to the reader to generalize), performed much better when they learned how to forecast the outcomes of their decision processes. TABLE 3 Stepwise regression result using the period profits as the dependent variable and the absolute values of the errors in the three forecasts and the existence of a cash shortfall as independent variables Unstandardized β Coefficients Standardized Beta “p” value Model β Constant 7,107 < 0.0005 Abs value of the error in the ROS forecast - 203,888 - 0.287 0.002 Abs. value of the error in cash forecast - 0.224 - 0.308 0.001 Chapter 11 condition 5,884 0.238 0.002 Abs. value of the error in forecasting EPS - 1,634 - 0.263 0.007 Model Summary Model R R2 Adjusted R2 4 independent variables 0.689 0.475 0.455 ANOVA Sums of Squares Df “p” value Regression 3.94E+09 4 <0.0005 Residual 3.60E+09 105 Total 7.03E+09 109 Developments in Business Simulation and Experiential Learning, Volume 34, 2007 65 Also, as the game went on, they learned about the process of managing a simulated firm. Is this learning transferable to the practitioner world? That is beyond the capabilities of this data set, but this author can’t help but believe that it is. IMPLICATIONS This analysis has implications regarding what simulation participants learn and the links between learning and their firms’ performance. It also has implications for game design. If one measures game performance by forecasting abilities rather than profits, then games can be designed in more realistic way. We tend to fool ourselves into believing that our simulations are realistic because they produce income statements and balance sheets. However, what actual industry has every firm start with equal assets and exactly equal opportunities at the margin for each decision variable? Students should learn to leverage whatever their firm’s inherent advantages may be. Some firms have better R&D facilities than others. Some firms have access to lower costs of capital. Some firms are better marketers than others. Some can manufacture products with lower variable costs and /or lower overhead than others, while still others have superior design capabilities. One thing is certain; an industry where all firms have equal opportunities never exists. If one measures simulation performance by the ability to forecast outcomes, the nature of business games becomes much more like the real world. Also – “end-play,” where teams try to “beat” the game during the last period would disappear. There would be no reason to make drastic strategy or decision changes while making a last ditch attempt at winning. In fact, it would be best not to make drastic changes because the results of the drastic changes would be more difficult to forecast. The author encourages other teachers, researchers and users of business simulations to experiment with using a variety of measures other than the firm’s financial outcomes to evaluate student performance in a business game. Although short run financial performance is used on Wall Street, it is not necessary to use the same indicators to measure student learning. REFERENCES _______ (2004) CAPSTONE, Business Simulation, Team Members Guide 2005, Management Simulations, Inc, Northfield, IL Arrow, Kenneth (1962), "The Economic Implications of Learning by Doing," Review of Economic Studies (June) Vol. 29, No. 3, pages 153 - 173 Anderson, Philip H, Lawton, Leigh (1992), “The relationship between financial performance and other measures of learning on a simulation exercise,” Simulation & Gaming (Sep) Vol. 23, No. 3; pages. 326 - 340 Anderson, Philip H, Lawton, Leigh (1992b), “A survey of methods used for evaluating student performance on business simulations,” Simulation & Gaming:(Dec) Vol. 23, No. 4; p. 490 – 498 Capon, Noel and Peter Palij (1994), ”Strategic marketing forecasting, market segmentation selection and firm performance,” International Journal of Forecasting (Sept.) Vol. 10, No. 2 pages 339 - 352 Cheisl, Newell (1987), “The use of a simple forecasting technique during an interactive computerized business game,” Developments in Business Simulation and Experiential Exercises, Vol. 14, pages 39 – 42 Ciccone, Stephen J (2005) “Trends in analyst earnings forecast properties,” International Review of Financial Analysis Vol. 14, No. 1 page 1 Ferguson, Miles D, (1992), “Forecasting as a learning tool,” The Journal of Business Forecasting Methods and Systems, Vol. 11, No. 3 (Fall) pp 27 – 30. Gentry, James W. and Edward T. Reutzel (1977), “Teaching inventory control via computer interaction with increasing complexity,” Computer Simulation and Learning Theory, Vol. 3 pages 219 - 228 Glazer, Rashi, Steckel, Joel H., Winer, Russell S (1989), “The Formation of Key Marketing Variable Expectations and Their Impact on Firm Performance: Some Experimental Evidence,” Marketing Science, (winter) Vol. 8, No. 1; pages. 18 – 35. Gosen, Jerry and John Washbush (2001), “An initial validity investigation of a test assessing total enterprise simulation learning,” Developments in Simulation and Experiential Exercises, Vol. 28 pages 92 – 95 Gosenpud, Jerry, Paul Miessing and Charles J. Milton (1984), “A research study on strategic decisions in a business simulation,” Developments in Simulation and Experiential Exercises, Vol.11pages 161 -165 Hommes, Cars, Joep Sonnemans, Jan Tuinstra and Henk van de Velden, (2005), “A strategy experiment in dynamic asset pricing,” Journal of Economic Dynamics & Control, Vol. 29 No. 4 (April) pages 823 – 833. Hoskin, Robert E., John S. Hughes, and William E. Ricks, (1986). Evidence on the incremental information content of additional firm disclosures made concurrently with earnings. Journal of Accounting Research, vol. 24 (Supplement), pages 1 - 32 Ittner, Christopher and David F Larcker (1998),” Are nonfinancial measures leading indicators of financial performance? An analysis of customer satisfaction,” Journal of Accounting Research Vol. 36, No. 1 pages 1 - 35 Jain, Chaman L. (2005/2006), “Benchmarking New Product Forecasting,” The Journal of Business Forecasting Vol. 24 No 4 Pages 16 to 17 LaFollette, William R. and James A. Belohlav (1981), “Using simulation to study the effects of team compatibility,” Journal of Experiential Learning and Simulation Vol. 3, No. 4 pages to 8 to 12 Landry, Steven P, Terrance Jalbert, and Canri Chan (2005),” Analyzing Functional Performance of Hong Kong Firms: Planning, Budgeting, Forecasting, and Developments in Business Simulation and Experiential Learning, Volume 34, 2007 66 Automation” Journal of American Academy of Business, (Sept.) Vol. 7, No. 1; pages. 228 to 332 Lapide, Larry (2005), “An S&OP Maturity Model,” The Journal of Business forecasting (fall) Vol. 24 No. 3 pages 15 to 21 Machuga, Susan M, Ray J. Pfeiffer, Jr. and Kiran Verna, (2002), “Economic value added, future accounting earnings and financial analysts earnings per-share forecasts,” Review of Quantitative Finance and Accounting (Jan) Vol. 18, No. 1; pages 59 - 73. 59 Mansfield, Edwin (1996), “A note on estimating the returns from new technology: How much learning?,” International Journal of Technology Management, Vol. 11, No 7 pp814 - 821 Pickett, Gregory M. and Roxanne Stell (1987), “Teaching forecasting to the masses, Developments in Business Simulation and Experiential Exercises, Vol. 14, pp 165 – 168 Render, Barry and Ralph. M. Stair, Jr. Quantitative Analysis for Management, Allyn and Bacon, 1982 Rotenberg, Wendy (1994), “Discussion of “voluntary inclusion of forecasts in the MD&A section of annual reports,” Contemporary Accounting Research (fall) Vol. 11 No. 1 pages 489 to 497 Shen, Weimimng, Francisco Maturana and Douglas H. Norrie (2000), “Enhancing the performance of an agent based manufacturing system through learning and forecasting,” Journal of Intelligent Manufacturing, Vol. 11 No. 4 pages 365 – 388 Skinner, Douglas J (1994), “Why firms voluntarily report bad news,” Journal of Accounting Research (spring) Vol. 32, No. 1 pages 38 - 61 Teach, Richard (1993) “Forecasting accuracy as a performance measure in business games.” Simulation & Gaming: (Dec) Vol. 24 No. 4 pages. 476 - 490 Varanelli, Andrew, Jr. and Susan M. Fazio (1981), “A case study in the use of experiential learning to enhance student understanding of strategy evaluation and policy formulation,” Developments in Business Simulation & Experiential Exercises, Vol. 8, pp. 184 – 194 Washbush, John and Jerry Gosen (2002), “Total Enterprise simulation learning compared to traditional learning in the business policy course,” Developments in Business Simulation & Experiential Exercises, Vol. 29, pp. 281 – 286 Washbush, John B., (2003), “Simulation performance and forecast accuracy – is that all?” Developments in Business Simulation & Experiential Exercises, Vol. 30, pp. 250 – 253 Table of Contents Volume 34, 2007 Experiential Teaching May Lead To Experiential Learning The Role Of Learning Versus Performance Orientations When Reacting To Negative Outcomes In Simulation Games: Further Insights An Analysis Of The Interaction Of Firm Demand And Industry Demand In Business Simulations Consistency Of Participant Simulation Performance Across Simulation Games Of Growing Complexity Assessing And Incentivizing Learner Contribution And Performance Excellence: Creating Win-Win Evaluation Options The Technological Impact Analysis: A Research-Based Exercise To Heighten Learners' Technological Sensitivity Beginnings: How We Start the Semester and Individual Classes : A Roundtable Discussion Simulation Performance and Its Effectiveness As A PBL Problem: A Follow-Up Study The Use of Multimedia Learning Tools to Facilitate Online Learning of Business Statistics Forecasting Accuracy And Learning: A Key To Measuring Business Game Performance Panel Discussion: Alternative Ways of Using the Internet for Business Simulations to Input Decisions, Process, and Present Financial and Economic Data Output Absel Research -- One Additional Perspective On Where We Are And Where We Have Come From Assessing participant learning in a business simulation From Case Presentation To Case Facilitation: How Assessment Changed The Capstone Course The Use Of Computer-Assisted, Interactive Role-Play Simulation In Hong Kong Corporate Positioning: A Business Game Perspective Demonstration Of A Computer-Assisted Global Business Simulation Applying .NET Remoting To A Business Simulation Assessing Emotional Intelligence: The EQ Matrix Exercise Outcomes And Observations Of An Extended Accounting Board Game The Use Of Learning Styles Questionnaire In Hong Kong Forming Teams For Classroom Projects Online Budgeting And Marketing Control With The Proforma Analysis Package The Effect Of Experiential Learning Experiences On Management Skills Acquisition Students' Perceptions on the Individual Managerial Performance Orderness: A New Definition Of Alignment An Exploratory Study Of The Efficacy Of Servsafe® Online Team Behavior And Team Success Results From A Board Game Simulation Telecommuting Internships - Do They Work? The WEE GAME: A Pre-Game Effects On Learning When Students Have Information About Games And Their Outcomes When Playing Them Assessment And Simulations: Measuring The Academic Learning Compacts Within Dangerous Business: An Interactive Ethics Case Activity The Paradise Islands Assessing And Applying fiAppropriatefl Conflict Management Styles Distance Learning In Communication: Blended Or On-Line? Developing An On-Line Advanced Communication Course World Café: Simulating Seminar Dialogues in a Large Class The Programming Game: An Exploratory Collaboration Between Business Simulation And Instructional Design The Canary Principle: An Alternative Model for Providing Real-Time Coaching in an On-Line Discussion Environment Assessing Character Development Using A Direct-Approach Business Ethics Exercise Persistence Of Decisions By Simulation Game Participants The Invalidity of Profit = f(Product Quality) PIMS Validation of Marketing Games A Study of The Applicability of The Perceptions of Organizational Politics Scale (POPS) For Use in The University Classroom The ROI Of Connected Projects: A Payroll Example Initiation of Research on Gaming Simulation in Japan Inappropriate Use of Citations and Corrupting the Body of Knowledge: Accepting Urban Legends as Truth An Exploration Of The Perceived Value Of Highly Socio-Inductive Learning How "Whole" Is Whole Person Learning? An Examination Of Spirituality In Experiential Learning Modeling Outsourcing and Strategy Alignment into a Business Game Successful Integration Of Webct Into A Small Business School The Thinking Steps Model In Game Theory: A Qualitative Approach In Following The Rules The Application Of Means-End Theory To Understanding The Value Of Simulation-Based Learning Armchair Travel: An active learning approach to increasing global awareness and participant self-efficacy Goldratt's Thinking Process: Is there a place for it in the Total Enterprise Simulation Assessing And Developing Student Skills Using a Group Exercise Computer Business Simulation Design: Novelty & Complexity Issues