Correlations of Measures of Forecasting Accuracy and Profit Page 183 - Developments in Business Simulation and Experiential Learning, Volume 44, 2017 ABSTRACT Forecasting is an integral function of business generally. Commensurate with this, developers and researchers of business games have allotted considerable conceptual and empirical attention to forecasting. Dickinson (2016) investigated intercorrelations of six measures of forecasting accuracy. That research is extended here with correlations of those measures with two measures of company profit. INTRODUCTION Forecasting is an integral function of business generally. Accordingly, forecasting has been the object of considerable research in business games. This attention has often been in the context of evaluating the performance of game participants. Across the many researches numerous operationalizations of forecasting accuracy have been employed. Individual contexts and individual game administrators may well call for different operationalizations. Yet it may be that some of these are more conceptually and empirically superior to others. With this in mind, Dickinson (2016) investigated the intercorrelations among six definitions of forecasting accuracy. Having estimated the degree to which the measures are not perfectly correlated he then, more on a conceptual and rationalized basis, recommended a single measure. Researchers and administrators using a common measure of forecasting accuracy would make more meaningful the integrating of research findings and perhaps inform the design of business games. Forecasting is a function of business in its own right. And forecasting accuracy need not necessarily be used as a criterion for evaluating game participant performance. An enduring discussion in business gaming, though, does position the suitability of forecasting accuracy versus profit for evaluating performance. The present research, then, extends Dickinson’s (2016) intercorrelations of measures of forecasting accuracy to include their correlations with company profit. Much of the foundation of Dickinson’s (2016) research is, of course, applicable to the present research. Elements of that foundation are presented here followed by the previous forecasting accuracy intercorrelations now augmented with profit correlations. Forecasting is prominent in business games. Of specific interest here is sales forecasting, although forecasts of several other criteria have been employed. (More exhaustive lists of criteria that have been forecasted may be found in deSouza, Bernard, & Cannon (2010, 20 criteria), Hand & Sims (1975, 6 criteria), and Teach (2007, 7 criteria).) Forecasting of unit sales has been used by Anderson & Lawton (1992b, 1990), de Souza, Bernard, & Cannon (2010), Gosen & Washbush (2001), Hand & Sims (1975), Palia (2011, 2004), Peach & Platt (2000), Teach (2007, 1990, 1989, 1987), Washbush (2003), and Wolfe (1993). Forecasting of market share has been used by Anderson & Lawton (1988), de Souza, Bernard, & Cannon (2010), Gosen & Washbush (2002), Teach (2007, 1990, 1989, 1987), and Wolf (1993). In turn, the use of profit to evaluate company performance seems widespread. Early on, a survey of business game users found 92.5 percent of respondents used “team performance versus other teams.” (Anderson & Lawton 1992a, pp. 493) In that article it is not absolutely clear that “performance” meant profit or profit related, e.g., return-on-investment, performance; this seems to never be explicated. From context, though, it appears that “performance” meant profit or profit-related performance. The article commences with a discussion that “financial performance on a business simulation exercise is best summarized by measures of net income, return on investments (ROI) and return on assets (ROA)...no real relationship between financial performance and learning...too much weight has been placed on profits...” (p. 490) And from Anderson & Lawton (1992b, p. 326), “Anderson & Lawton (1989) reported 100% of those surveyed used financial performance as one determinant of a student’s grade.” Teach began his seminal article on the use forecasting for evaluation with, “The vast majority of faculty who use business simulations...seem to evaluate a team’s performance based on some function of the profit...the simulated firm has accumulated...” and “Many other articles, too numerous to mention here, refer to ‘team performance’ as a euphemism for cumulative profits...” (1990, pp. 12-13) PREVIOUS STUDIES Conceptually, making profit and making accurate sales forecasts are not necessarily the result of the same managerial abilities. And at the core of the profit~forecasting discussion is which best reflects learning and learning in what respects. As to whether the two types of company performance criteria nevertheless result in comparable evaluations, empirical studies are equivocal. Involving forecasting a mix of criteria–unit market share, unit sales, cash flows, profit-loss–Teach’s (1989) study, “...clearly indicates that in a simulated environment, there is a very strong relationship between the ability of a simulation team to forecast and the relative profitable [sic] of the firm which the team manages...measuring forecast abilities of the management team may be substituted for direct profitability measures.” (p. 106) Hand & Sims (1975) found the correlation between sales forecasting error and net earnings to be statistically significant (r=–0.27, p=.01) as was the correlation of forecasting error and return on investment (r=-0.28, p=.01). Wolfe (1993, p. 53) reports, “...in a recent paper by Smith & Golden (1991)...It was found that nonsignificant relationships existed between a team’s forecasting accuracy and its economic performance when measured as profits and short-term and long- term stock prices.” Starkly, Anderson & Lawton (1992b, p. 334) found the correlation between the accuracy of forecasting unit sales and a composite score based on net income, return on investment, and return on assets to equal zero. The present research complements and extends these Correlations of Measures of Forecasting Accuracy and Profit John R. Dickinson University of Windsor MExperiences@bell.net mailto:Mexperiences@bell.net Page 184 - Developments in Business Simulation and Experiential Learning, Volume 44, 2017 previous studies and that of Dickinson (2016). In the former vein, sales forecasting accuracy (actually error) is correlated with two indicators of company profit. Extending those previous studies, the correlations here involve multiple operationalizations of forecasting accuracy (error). FORECASTING DEMAND (NOT SALES) Dickinson (2013, p. 102) has noted that in “sales” forecasting it is not actually sales that is forecasted, but demand. ...it is useful to clarify that “demand” refers to product units that consumers seek to purchase. This is the demand attributable to the desirability to consumers of the company’s offering (along with other factors, e.g., the game environment structure and competitors’ strategies). In contrast, “sales” refers to units actually sold. The difference between demand and sales is stockouts. Stockouts reflect the opportunity loss of sales that could have could have been made, but were not due to lack of availability. It is demand that is influenced by company strategy. Sales equals demand where sufficient units are available. Accordingly, Dickinson (2016) investigated correlations among measures of forecasting accuracy based on sales that have been used in business gaming research plus corresponding measures based on demand. IMPLICIT FORECAST An implicit forecast is utilized here. Following Dickinson (2013), consider games where managers order product units for resale or produce units for sale. The number of units the manager makes available for sale–beginning inventory plus units ordered for resale–is taken to be his or her forecast. (There is nothing in the configuration of the specific game, e.g., supply disruption, inflation in supplied unit price, etc., that would support this not being the case.) Where a stockout occurs, the forecast error is the amount of the stockout. Where a stockout does not occur, the forecast error is the amount of ending inventory. The implied unit sales forecast is the number of product units ordered or produced plus any inventory available at the beginning of the competition period. When a manager does not order/produce units, presumably the manager anticipates, i.e., forecasts, beginning inventory to be sufficient. “Sufficient” is an inexact forecast, of course, and in such cases no implicit forecast value is available. In the present study, where 1604 product orders might have been placed, in only 5.4 percent of them was an order not placed. Above, the critical distinction between demand and sales is explained. Where a stockout occurs, unit sales are not a valid indicator of the consequences of a company’s (marketing) strategy. Essentially, in such cases, the consequences are understated by sales and the calculation of forecasting accuracy (error) using unit sales is erroneous. In the present study, both sales and demand are present among the measures of forecasting accuracy. Where a stockout does not occur, unit demand is equal to unit sales. Where a stockout does occur, demand is equal to unit sales plus the unit amount of the stockout. MEASURES OF FORECASTING ACCURACY (ERROR) Common nomenclature notwithstanding, measures of forecasting accuracy are, more accurately (!), measures of forecasting error. Where S=unit sales, D=unit demand, and F= (implicitly) forecasted unit demand, the measures of forecasting accuracy (error) investigated here are: [1] |S-F|/S [2] |D-F|/D [3] |S-F|/F [4] |D-F|/F [5] |S-F| [6] |D-F| Though all six measures are studied here, based on considerations (1) that it is demand that is actually forecasted (i.e., [2], [4], [6]), TABLE 1 MEAN INTERCORRELATIONS AMONG MEASURES OF FORECASTING ACCURACY [1] [2] [3] [4] [5] Measure |S-F|/S |D-F|/D |S-F|/F |D-F|/F |S-F| [2] |D-F|/D .889 [3] |S-F|/F .975 .844 [4] |D-F|/F .667 .880 .650 [5] |S-F| .935 .813 .943 .618 [6] |D-F| .748 .899 .728 .874 .799 Entries are means for 174 correlations. Page 185 - Developments in Business Simulation and Experiential Learning, Volume 44, 2017 (2) relative measures of accuracy (i.e., [1]-[4]) are more comparable than absolute measures across markets and business games of varying size and, being so, also facilitate the accumulation of knowledge regarding forecasting accuracy across different researches, and (3) demand being the object of the forecast and not vice versa and demand, founded in the (simulated) market being a more fundamental basis than the more idiosyncratic forecasts of individual managers Dickinson (2016) recommended that |D-F|/D be considered for general adoption by game designers and researchers. INTERCORRELATIONS OF SIX MEASURES OF FORECASTING ACCURACY Dickinson (2016, p. 132) presents intercorrelations among the six measures of forecasting accuracy. Those correlations are for each of four product-market segments separately in which the particular game companies might operate. To summarize those correlations, which are but background here, Table 1 below presents those correlations for all four segments together. DATA Facilitating the extension of Dickinson’s (2016) intercorrelations among forecasting accuracy measures, the same data utilizing The Marketing Management Experience (MME, Dickinson 2006) business game are analyzed here. Added to the (implicit) forecast, sales, and demand data are two measures of company profit: operating income and after tax earnings. In the MME taxes are levied for the company, not for individual region-product segments. That total company tax is then allocated to segments on the basis of the segments’ respective unit sales. In like fashion, any company losses are carried over to the next competition period and are allocated to segments on the basis of unit sales. Operating income is calculated prior to other income (i.e., interest), loss carryover, and taxes and, in sum, is a more direct measure of segment profitability, while segment after tax earnings is a more “full cost” measure of profit. In the MME, companies may operate in either or both of two geographic regions and may market either or both of two products (a digital still camera and a digital video camera), giving rise to four region-product segments. For each of the four segments, an order for products to be resold may be placed with an inventory being maintained in each. For each competition period, then, there are potentially four implicit forecasts for each company. (MME managers need not necessarily operate in all four segments in any given period, though most choose to do so.) Data were potentially available for 51 companies in each of the four region-product segments. Companies competed within 12 industries for nine periods, following a single trial period. As noted above, an implicit forecast cannot be calculated when a product order is not placed in a given competition period. Too, again, managers may choose to not operate in some segments. The actual number of data points, then, was reduced accordingly. Table 2 presents the number of implicit sales forecasts for each MME region-product segment. Also in Table 2 are the number of instances where an inventory stockout occurred. As explained above, stockouts are a critical consideration in operationalizing the consequences of a company’s strategy; where a stockout occurs, sales is not an accurate measure of the effect of strategy. ANALYSIS AND RESULTS For each company and each region-product segment and each competition period an implicit sales forecast was calculated as described earlier. Corresponding unit sales (affected by stockouts) and unit demand (unaffected by stockouts) were known. For a given company and segment, then, for each period a measure of forecasting accuracy was calculated. This was done for each of the six measures defined above. Plus, of course, operating income and after tax earnings were available for each segment for each period. Across the nine competition periods, then, a correlation between a given accuracy measure and a given profit measure was calculated. This was done “within company.” (To ensure stability, a correlation was only calculated when data were available for at least six of the nine periods.) Thus, for each company a correlation between each combination of forecast and profit measures was calculated. Means of these correlations were calculated across the companies; 39-46 companies depending on the specific region- product segment. Those mean correlations are presented in Table 3. INTERPRETATION Per consideration (2) under “Measures of Forecasting Accuracy (Error)” above, correlations based on the absolute forecasting accuracy measures, |S-F| and |D-F| are discounted here and interpretations made only for the relative measures, i.e., [1]-[4]. DEMAND VERSUS SALES As explained earlier, it is unit demand that is forecasted by managers, not unit sales. In Table 3 it is correlations based on demand that are greater than their counterparts based on sales. Forecast accuracy (error) measures [1] and [2] differ with TABLE 2 NUMBERS OF FORECASTS AND STOCKOUTS Region-Product Segment Number of Forecasts Number of Stockouts Stockouts as % of Forecasts 1 2 3 4 381 321 401 379 80 32 127 72 21.00 9.97 31.67 19.00 Page 186 - Developments in Business Simulation and Experiential Learning, Volume 44, 2017 respect to the former being based on sales and the latter being based on demand. Across the four market segments and pertaining to both operating income and after tax earnings the latter mean correlations, i.e., |D-F|/D, are all greater in absolute value than the former correlations, i.e., |S-F|/S. A similar comparison can be made between measures [3] and [4]. And again, correlations of the measure based on demand, i.e., |D-F|/F, are all greater in absolute value than the correlations of the measure based on sales, i.e., |S-F|/F. Each row of correlations in Table 3 comprises results across the six measures of forecasting accuracy. Another consistent result with respect to accuracy measures based on relative demand, i.e., [2] and [4], is that across all four segments and both of the profit measures these relative demand correlations are the two highest within each row. TABLE 3 Mean Correlations Between Forecasting Error Measures and Profit (Standard Deviations) [1] [2] [3] [4] [5] [6] |S-F|/S |D-F|/D |S-F|/F |D-F|/F |S-F| |D-F| SEGMENT 1 (n=45 companies) Operating Income -.212 * (.477) -.331 (.396) -.185 (.487) -.396 (.361) -.157 (.473) -.253 (.397) After Tax Earnings -.118 (.450) -.199 (.400) -.093 (.456) -.262 (.393) -.085 (.423) -.159 (.369) SEGMENT 2 (n=39 companies) Operating Income -.279 (.390) -.315 (.378) -.287 (.407) -.339 (.387) -.215 (.425) -.236 (.413) After Tax Earnings -.242 (.370) -.271 (.355) -.259 (.381) -.290 (.361) -.275 (.336) -.287 (.309) SEGMENT 3 (n=46 companies) Operating Income -.211 (.429) -.439 (.316) -.189 (.435) -.455 (.322) .-109 (.446) -.310 (.389) After Tax Earnings -.215 (.447) -.456 (.333) -.193 (.450) -.482 (.335) -.113 (.455) -.333 (.388) SEGMENT 4 (n=44 companies) Operating Income -.429 (.313) -.457 (.300) -.410 (.328) -.456 (.295) -.270 (.348) -.266 (.345) After Tax Earnings -.431 (.331) -.459 (.314) -.416 (.346) -.457 (.298) -.279 (.351) -.272 (.343) * In market Segment 1, of the 45 correlations available for 51 companies, the mean correlation between the |S-F|/S forecasting accuracy measure and operating income is -0.212. The standard deviation of those correlations is 0.477. That all the correlations are negative is due to the forecasting measures actually being measures of error rather than accuracy. Page 187 - Developments in Business Simulation and Experiential Learning, Volume 44, 2017 Anderson, Philip H. & Lawton, Leigh (1992a). A survey of methods used for evaluating student performance on business simulations. Simulation & Gaming, Vol. 23, No. 4 (December), 490-498. Anderson, Philip H. & Lawton, Leigh (1992b). The relationship between financial performance and other measures of learning on a simulation exercise. Simulation & Gaming, Vol. 23 (September), 326-340. Anderson, Philip H. & Lawton, Leigh (1990). The relationship between financial performance and other measures of learning on a simulation exercise. In Wingender, John R. & Wheatley, Walt J. (Eds.), Developments in business simulation & experiential exercises, Volume 17 (Association for Business Simulation and Experiential Learning), 6-10. Reprinted in The Bernie Keys Library, available at www.absel.org. Anderson, Philip H. & Lawton, Leigh (1988). Assessing student performance on a business simulation exercise. In Sanders, Patricia & Pray, Thomas (Eds.), Developments in business simulation and experiential exercises, Volume 15 (Association for Business Simulation and Experiential Learning), 241-245. Reprinted in The Bernie Keys Library, available at www.absel.org. de Souza, Moises Pacheco, Bernard, Ricardo R. S., & Cannon, Hugh M. (2010). Another look at the use of forecasting accuracy on the assessment of management performance in business simulation games. In Smith, J. Alexander (Ed.), Developments in business simulation and experiential learning, Volume 37 (Association for Business Simulation and Experiential Learning), 68-78. Reprinted in The Bernie Keys Library, available at www.absel.org. REFERENCES STANDARD DEVIATIONS Again comparing the relative sales based forecast accuracies with their demand based counterparts, in all instances the standard deviation of the correlations is greater for the former than for the latter. With the latter being more accurate, this result underscores the greater consistency of demand based forecasts. PROFIT MEASURES As explained earlier, in the MME, operating income in each of the four segments is a more direct measure of segment profitability than is after tax earnings. It might be expected, then, that correlations involving operating income would be higher in absolute value than those involving after tax earnings. This is the case for Segments 1 and 2. However, the reverse is the case for Segments 3 and 4, though the differences in mean correlations are very small. Operating income in the MME is a better measure of segment profitability, but not purely so. Company-wide costs, e.g., fixed and marketing research, are allocated to segments. Research and development costs are for each of the two products and retail outlet costs are for each region and these are allocated to segments. (Again, the two products and two regions give rise to the four product-region segments.) DISCUSSION Inconclusiveness attends the profit~forecasting conversation. Fundamentally, which basis better reflects learning begs the issue of measuring true learning. As is the case in many validity contexts, were psychometricians able to actually measure true learning there would be no essential role for either profit or forecasting to act as operational definitions. Empirically, it could hardly get any more inconclusive than Teach’s, “...measuring forecast abilities of the management team may be substituted for direct profitability measures” (1989, p. 106) versus Anderson & Lawton’s (1992b, p. 334) zero correlation between the two. Teach’s (1990) and Wolfe’s (1993) seminal exchange brings both theoretical and empirical factors to the discussion. Despite that, along with numerous ensuing studies, what should be the relationship between the two remains inconclusive. Normatively, should the two be highly correlated or should they be uncorrelated or, as in the present study, should they be of substantial though not extreme correlation? If there is merit in high correlations, here the two measures of accuracy incorporating demand (rather than sales) which are also relative dominate the other four measures. This reinforces Dickinson’s (2016) recommendation of the |D-F|/D measure, though here the |D-F|/F measure fares slightly better. Hand & Sims (1975) found the respective correlations of net earnings and return on investment with sales forecasting error to be nearly equal (-0.27 and -0.28). Here, for the two measures of profit employed–operating income and after tax earnings–there is no consistent pattern of correlations across the four product-market segments. For Segment 1 the respective correlations with forecast accuracy are quite different. For Segments 3 and 4 the respective correlations are nearly equal. Forecasting accuracy and profit generation are both important, obviously, to the prosperity–even survival–of businesses. The initial propositions of using forecasting accuracy as a measure of manager performance in business games and the immediately ensuing reinforcements of profit as a measure took positions of either-or. The two measures, though, are both conceptually and operationally different and there seems to be no reason why one measure must be adopted to the exclusion of the other. One implication of this is that even where the two criteria are empirically highly correlated their manifest values derive from respective underlying morasses that are fundamentally different. Yet both forecasting accuracy and profit amount are sterile criteria. In and of themselves they provide no insight into the information, thought processes, decisions, and analytical methods employed by managers resulting in either. In short, they themselves have no diagnostic capacity. Neither has any delineable connections to elements comprising its underlying morass. This limits the student learning that is the overriding objective of business games. This lack was the major impetus for Kaplan & Norton’s Balanced Scorecard; i.e., to complement traditional financial measures of performance: “...measures that tell the results of actions already taken...[rather than] ...operational measures that are the drivers of future financial performance.” (1992, p. 71) Whatever their capacity-for-learning limitations, both forecasting accuracy and profits are common measures of company performance and are on their face simple values. These may be sufficient for some applications of business games. Alternatively, business game researchers have not been neglectful in adopting variations of the Balanced Scorecard as well as similar multi-criteria evaluation schemes. http://www.absel.org./ http://www.absel.org./ http://www.absel.org./ Page 188 - Developments in Business Simulation and Experiential Learning, Volume 44, 2017 Dickinson, John R. (2016). Intercorrelations of measures of forecasting accuracy and a recommendation. In Developments in business simulation and experiential learning, Volume 43 (Association for Business Simulation and Experiential Learning), 130-135. Reprinted in The Bernie Keys Library, available at www.absel.org. Dickinson, John R. (2013). An implicit measure of forecasting accuracy. In Boscia, Marian (Ed.), Developments in business simulation and experiential learning, Volume 40 (Association for Business Simulation and Experiential Learning), 102-106. Reprinted in The Bernie Keys Library, available at www.absel.org. Dickinson, John R. (2006). The marketing management experience. Windsor, Ontario: Management Experiences. ISBN: 0-9691231-1-6 Gosen, Jerry & Washbush, John (2002). The validity investigation of a test assessing total enterprise simulation learning. In Vaughan, Mary Jo & Pillutla, Sharma (Eds.) Developments in business simulation and experiential learning, Volume 29 (Association for Business Simulation and Experiential Learning), 247-251. Reprinted in The Bernie Keys Library, available at www.absel.org. Gosen, Jerry & Washbush, John (2001). An initial validity investigation of a test assessing total enterprise simulation learning. IPittenger, Khushwant S. & Vaughan, Mary Jo (Eds.), Developments in business simulation and experiential learning, Volume 28 (Association for Business Simulation and Experiential Learning), 92-95. Reprinted in The Bernie Keys Library, available at www.absel.org. Hand, Herbert H. & Sims, Henry P., Jr. (1975). Statistical evaluation of complex gaming performance. Management Science, Vol. 21, No. 6 (February), 708-717. Kaplan, Robert S. & Norton, David P. (1992). The balanced scorecard–measures that drive performance. Harvard Business Review, Vol. 70 (January-February), 71-79. Palia, Aspy (2011). Tracking forecast error type, frequency and magnitude with the forecast error package. In Murph, Elizabeth (Ed.), Developments in business simulation and experiential learning, Volume 31 (Association for Business Simulation and Experiential Learning), 45-58. Reprinted in The Bernie Keys Library, available at www.absel.org. Palia, Aspy (2004). Online sales forecasting with the multiple regression analysis data matrices package. In Feinstein, Andrew Hale & Potosky, Denise (Eds.), Developments in business simulation and experiential learning, Volume 31 (Association for Business Simulation and Experiential Learning), 180-182. Reprinted in The Bernie Keys Library, available at www.absel.org. Peach, E. Brian & Platt, Richard G. (2000). Total enterprise simulations and optimizing the decision set: assessing student learning across decision periods. In Page, Diana & Snyder, LT (Eds.), Developments in business simulation and experiential learning, Volume 27 (Association for Business Simulation and Experiential Learning), 242-247. Reprinted in The Bernie Keys Library, available at www.absel.org. Teach, Richard D. (2007). Forecasting accuracy and learning: a key to measuring simulation performance. In Smith, J. Alexander (Ed.), Developments in business simulation and experiential learning, Volume 34 (Association for Business Simulation and Experiential Learning), 57-66. Reprinted in The Bernie Keys Library, available at www.absel.org. Teach, Richard D. (1990). Profits: the false prophet in business gaming. Simulation & Gaming, Vol. 21 (March), 12-26. Teach, Richard D. (1989). Using forecasting accuracy as a measure of success in business simulations. In Pray, Tom & Wingender, John (Eds.), Developments in business simulation & experiential exercises, Volume 16 (Association for Business Simulation and Experiential Learning), 103-107. Reprinted in The Bernie Keys Library, available at www.absel.org. Teach, Richard D. (1987). Profits: the false prophet. In Kelly, Lane & Sanders, Patricia (Eds.), Developments in business simulation and experiential exercises, Volume 14 (Association for Business Simulation and Experiential Learning), 205-207. Reprinted in The Bernie Keys Library, available at www.absel.org. Washbush, John B. (2003). Simulation performance and forecast accuracy–is that all? In Pillutla, Sharma & Feinstein, Andrew Hale (Eds.), Developments in business simulation and experiential learning, Volume 30 (Association for Business Simulation and Experiential Learning), 250-253. Reprinted in The Bernie Keys Library, available at www.absel.org. Wolfe, Joseph (1993). On the propriety of forecasting accuracy as a measure of team management ability: a preliminary investigation. Simulation & Gaming, Vol. 24 (March), 47- 62. http://www.absel.org/ http://www.absel.org./ http://www.absel.org./ http://www.absel.org./ http://www.absel.org./ http://www.absel.org./ http://www.absel.org./ http://www.absel.org./ http://www.absel.org./ http://www.absel.org./ http://www.absel.org./