An Implicit Measure of Forecasting Accuracy Page 102 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 ABSTRACT Sales forecasting, and the accuracy thereof, is prominent in business games. Invariably, explicit forecasts are made by game participants. The present research, though, describes an implicit measure of forecasting accuracy. The measure has the benefit of being available retroactively for use, primarily, in basic research. A complement questions measuring forecasting accuracy without taking into account stockouts. INTRODUCTION AND PURPOSE Forecasting, more specifically sales forecasting accuracy, is prominent in a variety of roles in business games:  Conceptual and empirical underpinnings for using forecasting accuracy as a (or the) criterion for evaluation of participants (Anderson & Lawton, 1992; Hand & Sims, 1975; Teach, 2007, 1993, 1990; Wolfe, 1993a, 1993b)  Applications using accuracy of sales forecasts as a criterion for evaluation of participants (Peach & Plant, 2000, Teach & Patel, 2007)  Post hoc student assessment of learning and “...the construction of a test bank of items designed to assess the degree to which learning takes place from playing a total enterprise simulation (Gosen & Washbush, 2001, p. 92; also Gosen & Washbush, 2002, Gosen & Scott, 2000, Gosen et al., 1999)  Development of a decision support system diagnosing forecast error (Palia, 2011) and an online tool for students to use for forecasting (Palia, 2004) Invariably, the sales forecast is made via–How else?– self-report by game participants: Anderson & Lawton ,1992, 1990; de Souza, Bernard, & Cannon, 2010; Gosen & Washbush, 2001; Hand & Sims, 1975; Peach & Platt, 2000, p. 245; Teach, 2007; Washbush, 2003; Wolfe, 1993. But there is an “else” that principally might be used in basic research and does not require explicit forecasts made by participants. This report describes how forecasting accuracy may be inferred from participant decisions and basic outcomes produced by the game algorithm design; an implicit measure of sales forecasting accuracy. DEMAND VERSUS SALES For the present research, and generally, it is useful to clarify that “demand” here refers to product units that consumers seek to purchase. This is the demand attributable to the desirability to consumers of the company’s offering (along with other factors, e.g., the game environment structure and competitors’ strategies). In contrast, “sales” refers to units actually sold. The difference between demand and sales is stockouts. Stockouts reflect the opportunity loss of sales that could have could have been made, but were not due to lack of product availability. It is demand that is influenced by company strategy. Sales equals demand where sufficient product units are available. The incidence of stockouts may not be inconsequential. Dickinson (2006b, pp. 234-235) reported that, “Across the (four) competitions between 20.9 and 27.3 percent of approximately 6,000 inventories stocked out.” Several studies do not make clear whether the actual sales variable used takes into account stockouts (Anderson & Lawton, 1990, de Souza, Bernard, & Cannon, 2010, Gosen & Washbush, 2001, Hand & Sims, 1975, Teach, 1989, Wolfe, 1993a). Simple as the schema is, it is critically important where sales forecasting accuracy is involved. While it may be nominally termed a “sales” forecast, it is demand that is forecasted. AN IMPLICIT MEASURE OF ACCURACY (ERROR) Forecast “accuracy” and forecast “error” describe the same phenomenon. Whether termed “accuracy” or “error” two pieces of information are required: the forecasted value and an actual value, i.e., actual demand. As explained above, the actual value should be demand, not sales. (Researchers differ in the calculation made using these two pieces of information. E.g., absolute value of difference between forecast and actual divided by actual (de Souza, Bernard, & Cannon, 2010, p. 72; Teach, 1989, p. 104), actual divided by forecast (Peach & Platt, 2000, p. 245), absolute value of difference between actual and forecast (Gosen & Washbush, 2001, p. 93; Washbush, 2003, p. 251).) “FORECASTED” DEMAND Consider games in which product orders (for resale by the game company) must be placed or production of units must be planned. A simple progression is that the number of units ordered or produced plus the number of units available in inventory at the beginning of the competition An Implicit Measure of Forecasting Accuracy John R. Dickinson University of Windsor MExperiences@bell.net mailto:Mexperiences@bell.net Page 103 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 period equals the number of units the company manager anticipates, i.e., forecasts, selling. For example, in The Marketing Management Experience (Dickinson, 2006a) there are incentives for not ordering more units than anticipated to be sold, i.e., inventory carrying costs and product obsolescence. (Any improvement in product quality applies to new product orders, not units in inventory.) And there are no incentives for carrying excess inventory, e.g., as a precaution against future availability or cost increases. Too, of course, there is the basic incentive of having available inventory to meet demand, i.e., to not stockout. Thus, the number of product units in beginning inventory plus the number of product units ordered or produced is taken to be the manager’s sales forecast. DEMAND Capturing actual demand for the implicit accuracy (error) measure calculation depends on the information available. In some fashion game software processes companies’ (marketing) strategy decisions (along with other factors) to arrive at the number of units demanded (before adjusting for stockouts). Most directly, then, this number of units demanded is the second piece of information needed to calculate forecasting accuracy. Whether those results are retained by the game software or, if retained, made available to instructors (or students) is another matter. But they easily could be retained and made available and game designers might consider incorporating this feature. These demand amounts may be used whether or not companies are responsible for ordering or producing product units. DEMAND = ACTUAL SALES For games where supply is not unlimited, i.e., product orders must be placed or production of units must be planned, the simplest outcome is that available inventory (=beginning inventory plus units ordered or produced) is (more than) sufficient to satisfy demand. In settling on the number of units to order or produce, the manager has overestimated demand. Forecast error in this scenario equals units remaining in inventory at the end of the competition period. DEMAND > ACTUAL SALES Where available inventory is not sufficient to satisfy demand then a stockout occurs, actual sales equals available inventory, and actual sales are less than demand. In lieu of originally calculated demand, stockout amounts may be available to the instructor and, perhaps, to participants (The Business Management Laboratory, Jenson, 1999; Palia, 2011 [Compete]; Marketer: A Simulation, Smith & Golden, 1987). Demand, then, equals beginning inventory plus units ordered or produced plus the stockout amount. As with demand above, stockout amounts could be made readily available by game designers. Some games provide for automatically at least partially reducing stockouts by transfers of different types (Intop, Thorelli and Graves, 1964) or expedited or overtime production (Compete, Faria, 2006; The Marketing Game!, Mason & Perreault, 2002; Marketing Simulation, Bush & Burns, 1991). These provisions come at a cost to the company and, of course, reduce stockout amounts. These reduction amounts, though, are generally known and need only be added to the calculation of actual demand. Whether demand is available directly or available indirectly with knowledge of stockout amounts, forecast error equals that demand minus units available (units available taken to be the manager’s forecast). USES BASIC RESEARCH The primary use for this implicit measure of forecast accuracy may be in basic research. Prospectively, instructors may not wish to require forecasts from students, yet wish to conduct basic research involving forecasting accuracy. Implicit forecasts might be available for this purpose. In like fashion, a major benefit may be that with this approach implied sales forecasts may be available retroactively. For games where demand is retained, e.g., The Marketing Management Experience (Dickinson, 2006a), it is possible to conduct basic research involving forecasting accuracy even when game participants did not make explicit sales forecasts. PARTICIPANT EVALUATION Forecasting accuracy, of course, is widely used as a criterion for evaluating game participant performance. The implicit forecast accuracy measure broached here might potentially be used in this capacity. The difference between demand and sales is product availability. The mere fact that demand, rather than sales, is to be used to define forecast accuracy in itself might move participants to be more mindful of product units ordered or produced. In turn, presumably participants must be fed back demand information. This has an interesting pedagogical implication for business games. In the real world, managers rarely know demand; they know sales. One of the “costs” of stocking out is that the actual effect of (marketing) strategies on consumers is not known. Estimating the effectiveness of strategy is very much compromised. Possible exceptions might be where a manager is on the retail site and might personally observe unsatisfied customers or where rainchecks are issued. But such surely are exceptions and are also inexact. So informing game participants of demand is unrealistic. (In business-to-business environments where business customers place orders, it is possible that unsatisfied demand is better known than in the retail marketplace.) On the other hand, it would be educational for participants to be better able to gauge the effect of their strategies. That is, it would be beneficially informative for participants in learning to formulate strategies. Finally, of course, it may be desirable for participants to learn forecasting methods, i.e., to make explicit forecasts. Page 104 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 The act of making explicit forecasts is educational in itself. It has been suggested that “...in at least business gaming situations, the engagement or practice of forecasting is more important than the forecast’s accuracy.” (Wolfe, 1993, p. 56 citing Smith and Golden, 1991) However, the implicit measure of forecasting accuracy might be used in conjunction with, rather than in place of, participant explicit forecasts. After all, how is the participant ever to know the actual accuracy of his/her forecasts? EXAMPLE A competition using The Marketing Management Experiment (MME, Dickinson, 2006a) may be used to illustrate the implicit measure of forecasting accuracy. The MME comprises four product-region market segments (two products marketed in two geographic regions). Each segment is different, specifically having different market response functions. Game participants need not market in all four segments, although normally most participants do. Product orders are placed in each segment and inventories are maintained in each segment. The particular competition comprised 26 companies (across six industries) with the competition lasting nine periods. Potentially, then, a total of 936 (=26x4x9) implicit forecasts might be made. That number is reduced to 805 by companies not marketing in all segments for some or all periods. As explained above an implied forecast can only be made when a manager places a product order in a given period for a given segment. (In not placing an order evidently the manager believes the number of units in beginning inventory is sufficient. But “sufficient” is an inexact “forecast.” The manager evidently anticipates demand to be less than [or equal to] beginning inventory, but by how much less is not available.) Across the 805 segment-periods an order was not placed 52 (6.5%) times. Of the 753 instances where an order was placed, units remained unsold in inventory at the end of the period 614 times. The manager’s implicit sales forecast was beginning inventory plus units ordered. Those numbers of unsold units are the implicit measure of forecast error. In 139 instances (18.46% of 753) the company stocked out. In those instances forecast error equaled the number of units demanded minus the number of units actually sold, i.e., the amount of the stockout. Calculating forecast error as |demand-forecast|/demand the median forecast errors for the four market segments were 0.296 (n=192), 0.405 (166), 0.197 (201), and 0.291 (194), respectively. Forecasting accuracy, explicit or implicit, is of little interest in and of itself. It is interesting only in conjunction with, say, implications for management strategy, specific forecasting techniques, basic research relating accuracy to other constructs, and possibly game participant evaluation. This example illustrates that the implicit measure may make available forecasting accuracy that might, in turn, be used in such ways. LIMITATIONS As explained above, the implicit measure of forecast accuracy cannot be applied where a company does not place a product order or plan a production quantity, i.e., where in effect the manager does not make a forecast. (The manager obviously “forecasted” that existing inventory would be sufficient, but that is an inexact implicit forecast.) This would seem to be a manageable limitation. The incidence of this is low. In the data used for illustration above only 6.5 percent of company-period-segments did not have an order placed. For student evaluation purposes, an average error can be calculated based on the instances where a product order was placed or production quantity planned. For basic research purposes, an average error might similarly be used or the instances of no product order being placed simply omitted from analyses. Against this, comparing actual sales with explicit self- reported forecasted sales would seem to be even more problematic where stockouts occur (though this may be corrected by incorporating stockout information where available to arrive at actual demand). SAFETY STOCKS Textbooks commonly note, if not prescribe, that inventory might be maintained at a higher level than anticipated, i.e., higher than forecasted sales. This as a precaution against unexpectedly high demand and the avoidance of stockouts. Where managers do, then, maintain safety stocks the accuracy of the implicit measure of forecast accuracy is compromised; product orders (coupled with beginning inventory) are greater than implicitly forecasted demand. The maintenance of safety stocks is but a recognition that the manager’s explicit or implicit forecast may be erroneous. This practicality is akin to the maxim that 80 percent of advertising is wasted; we just don’t know which 80 percent. Thus managers continue to “overspend” on advertising. Nonetheless, normatively the manager’s aim should be to get the forecast correct and thereby eliminate both out of pocket costs and opportunity losses. LEVEL OF FORECAST ERROR Units of products ordered or produced are the basis for implied forecasts. These decisions are important in their own right. That game participants are not required to make explicit sales forecasts should not diminish the importance of order/producing decisions. Possible pedagogical benefits of requiring explicit sales forecasts are recognized above. Actively rather than passively making forecasts may lead to more accurate forecasts, e.g., the employment of forecasting tools. That, however, is an issue of the desirability of requiring explicit forecasts. The validity of implied forecasts does not rest on this consideration. Page 105 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 REFERENCES Anderson, Philip & Lawton, Leigh (1992). A survey of methods used for evaluating student performance on business simulations. Simulation & gaming, Vol. 23 (December), 490-498. Anderson, Philip H. & Lawton, Leigh (1992). The relationship between financial performance and other measures of learning on a simulation exercise. Simulation & gaming, Vol. 23 (September), 326-340. Anderson, Philip H. & Lawton, Leigh (1990). The relationship between financial performance and other measures of learning on a simulation exercise. In John R. Wingender & Walt J. Wheatley, Walt J. (Eds.), Developments in Business Simulation & Experiential Exercises, 17, 6-10. (Reprinted from Bernie Keys Library.) Bush, Ronald F. & Burns, Alvin C. (1991). Marketing simulation (3rd ed.). New York: HarperCollins Publishers. ISBN: 0-06361075-2 de Souza, Moises Pacheco, Bernard, Ricardo R. S., & Cannon, Hugh M. (2010). Another look at the use of forecasting accuracy on the assessment of management performance in business simulation games. In J. Alexander Smith, J. (Ed.), Developments in Business Simulation and Experiential Learning, 37 68-78. (Reprinted from Bernie Keys Library). Dickinson, John R. (2006a). The marketing management experience. Windsor, Ontario: Management Experiences. ISBN: 0-9691231-1-6 Dickinson, John R. (2006b). The invalidity of profit=f (market share) PIMS “validation” of marketing games. In J. Alexander Smith (Ed.), Developments in Business Simulation and Experiential Learning, 33, 232-237. (Reprinted from Bernie Keys Library.) Faria, A. J. (2006). Compete (5th ed.). University of Windsor. Gentry, James W., Commuri, Suraj R., Burns, Alvin C., & Dickinson, John R. (1998). The second component to Experiential learning: a look back at how ABSEL has handled the conceptual and operational definitions of learning. In John K. Butler, Jr., Nancy H. Leonard, & Sandra W. Morgan (Eds.), Developments in Business Simulation and Experiential Learning, 25, 62-68. (Reprinted from Bernie Keys Library.) Gosen, Jerry & Washbush, John (2002). The validity investigation of a test assessing total enterprise simulation learning. In Mary Jo Vaughan & Sharma Pillutla (Eds.) Developments in Business Simulation and Experiential Learning, 29, 247-251. (Reprinted from Bernie Keys Library.) Gosen, Jerry & Washbush, John (2001). An initial validity investigation of a test assessing total enterprise simulation learning. In Khushwant S. Pittenger & Mary Jo Vaughan (Eds.), Developments in Business Simulation and Experiential Learning, 28, 92-95. (Reprinted from Bernie Keys Library.) Gosen, J, Washbush, J., Patz, A., Scott, T. W., Wolfe, J, & Cotter, D. (1999). A test bank for measuring total enterprise simulation learning. Developments in Business Simulation & Experiential Exercises, 25, 82- 92. (Reprinted from Bernie Keys Library.) Gosen, J., Washbush, J., & Scott, T. W. (2000). Initial data on a test bank assessing total enterprise simulation learning. In Diana Page & LT Snyder (Eds.), Developments in Business Simulation & Experiential Exercises, 27, 166-171. (Reprinted from Bernie Keys Library.) Hand, Herbert H. & Sims, Henry P., Jr. (1975). Statistical evaluation of complex gaming performance. Management science, Vol. 21, No. 6 (February), 708- 717. Jensen, Ron (1999). The business management laboratory (revised 5th ed.). Little Rock, AR: Micro Business Publications. Mason, Charlotte H. & Perreault, William D., Jr. (2002) The marketing game! (3rd ed.). Boston: McGraw-Hill Irwin. ISBN 0-256-13988-1 Palia, Aspy (2011). Tracking forecast error type, frequency and magnitude with the forecast error package. In Elizabeth Murf (Ed.), Developments in Business Simulation and Experiential Learning, 37, 45-58. (Reprinted from Bernie Keys Library.) Palia, Aspy (2004). Online sales Forecasting with the multiple regression analysis data matrices package. In Andrew Hale Feinstein & Denise Potosky (Eds.), Developments in Business Simulation and Experiential Learning, 31, 180-182. (Reprinted from Bernie Keys Library.) Smith, Jerald R. & Golden, Peggy A. (1991). An unexpected relationship between forecasting and performance, working paper. University of Louisville. Smith, Jerald R. & Golden, Peggy A. (1987). Marketer: a simulation (2nd ed.). Boston: Houghton Mifflin Company. ISBN: 0-395-42546-8 Teach, Richard D. (2007). Forecasting accuracy and learning: a key to measuring simulation performance. In J. Alexander Smith (Ed.), Developments in Business Simulation and Experiential Learning, 34, 57-66. (Reprinted from Bernie Keys Library.) Teach, Richard D. (2006). Forecasting accuracy and earning: the key to measuring simulation performance. In J. Alexander Smith (Ed.), Developments in Business Simulation and Experiential Learning, 33, 48-57. (Reprinted from Bernie Keys Library.) Teach, Richard D. (1993). Forecasting and management ability: a response to Wolfe. Simulation & gaming, 24 (March), 63-72. Teach, Richard D. (1990). Profits: the false prophet in business gaming. Simulation & gaming, 21 (March), 12-26. Teach, Richard D. (1989). Using forecasting accuracy as a measure of success in business simulations. In Tom Pray & John Wingender (Eds.), Developments in Business Simulation & Experiential Exercises, 16, 103 -107. (Reprinted from Bernie Keys Library.) Page 106 - Developments in Business Simulation and Experiential Learning, volume 40, 2013 Teach, Richard D. (1987). Profits: the false prophet. In Lane Kelly and Patricia Sanders (Eds.), Developments in Business Simulation and Experiential Exercises, 14, 205-207. (Reprinted from Bernie Keys Library.) Teach, Richard D. & Patel, Vishal (2007). Assessing participant learning in a business simulation. In J. Alexander Smith (Ed.), Developments in Business Simulation & Experiential Exercises, 34, 76-84. (Reprinted from Bernie Keys Library.) Thorelli, Hans B. & Graves, Robert L. (1964). International operations simulations. New York: Free Press. Washbush, John B. (2003). Simulation performance and forecast accuracy–is that all? In Sharma Pillutla & Andrew Hale Feinstein (Eds.), Developments in Business Simulation and Experiential Learning, 30, 250-253. (Reprinted from Bernie Keys Library.) Wolfe, Joseph (1993a). On the propriety of forecasting accuracy as a measure of team management ability: a preliminary investigation. Simulation & gaming, 24 (March), 47-62. Wolfe, Joseph (1993b). Forecasting and management ability: rebuttal to Teach’s response. Simulation & gaming, 24 (March), 73-75. Table of Contents Volume 40, 2013 Leo Investor SERIOUS PLAY: ON-LINE PEPULATOR SERIOUS PLAY:ON-LINE GAME "MANAGE-ART" The Game Within the Simulation Game the Research Method Concept and Project With Game Design Implementation Managing Human Resources Simulation Rising Stars: A Study of Creativity as Experiental Learning in County Government Why do Gen Y Students Study Abroad? The Relationship Between Individual Growth and the Intent to Study Abroad Our Favorite Assignment: The In-Basket Activity Mirror-Mirror: How Can We Better Understand Our Consumption Behavior? Reflective Thinking, Writing and Imagery as Tools to Better Understand Abstract Marketing Concepts Increasing Intuitive Decision Making Speed and Accuracy by Further Understanding Intuitive Decision Making Using Emotional Means Group Exams: Are they Relevant and Reliable as a learning Tool? Integrative Learning: Exploring Opportunities in Business Simulations Simulated Tabletop Exercise for Risk Management - Anti Bio-Terrorism Multi Scenario Simulated Tabletop Exercise Simulated Tabletop Exercise for Electric Power Saving Management in a Small Organization Merging the Case Method and Simulation in Management Education: Is it Possible? Experiential Learning in Accelerated Human Resource Management Courses Digital Game Building as Assessment: A Study of Secondary Students' Experience Using an Investment Fund Simulation Integrated with a Business Game Creating Simulation Conditions Based On Real World Data Simulation Games in Training New Management Methods A Summary Overview of Cultural Differences in Higher Education Innovations And Future Directions In Education: Case Review For Best Online Teaching Practices An Implicit Measure of Forecasting Accuracy Immediate Feedback and Assessment Technique (IF-AT) testing forms: An overview of the tool and uses Teaching Secondary Mathematics: Pre-service Teachers' Digital Game Design, Pedagogy and 21st Century Skills Follow The Leader II Business Simulations: From Punch Cards to Web-Based University Engagement: Good Neighbors Budget Allocation Exercise Customizing Business Simulations: An Exploration and a Hierarchy A Review of the Simulation Research in the Academy of Management Journal: Suggestions for Strengthening the Research Conducted by ABSEL Members ABSEL Reflections: 40 Years Of Excellence, Now Going Forward How Many Options do Multiple-Choice Questions Really Have? The Management/Accounting Simulation Now 100% Web-based Design, Server Side Programming, and Implementation A Large-Scale Game Test of Early-Determined Game Finishes Large-Scale Business Games for Assurance of Learning Purposes The Gamification Of Education OB Simulation: A Hands-On Demonstration Converting Simulations for the Online Environment: The New Ginseng Game Repositioning Brands with the Web-Based Product Positioning Map Graphics Package Implications of Regulatory Focus Theory for Simulation and Experiential Learning Riskware: A Game for Teaching Software Project Risk Management Teamwork for Decision-Making Through Games: The Case of On-Line "Manage-Art" Virtual Pepulator, a Model for Teaching Negotiation Small-Scale Business Games for Assurance of Learning Purposes Incorporating Intellectual Property Issues into a Business Simulation© Nationalization and Privatization in a Computer-Assisted Business Game An Investigation of the Relationship of Plan Quality, Forecast Accuracy and Earnings Performance Under Equal and Unequal Starting Positions in Marketing Positions in Marketing Simulations Measuring the Performance Ranking Curve in Marketing Simulation Games The Effectiveness of SDM Method in Business Simulation Game 40 years of Brazilian S&G - Analysis and Perspectives Accounting For Externalities: Harnessing The "Face-In-The-Mirror" Phenomenon The Role Of Simulations In Organizational Learning: Building Individual Absorptive Capacity Assessing Project Management as an Academic Learning Outcome (ALO) At the Inflection Point: Designing Army Assessments "Economics in Practice" A Simulation Game for High School Students Teaching the Basics of Economics and Entrepreneurship Fueling the Force: Exploring Leader Priorities Implementing Mental Models: Extending Insight and Whole Person Learning Using Business Simulations To Introduce Business Concepts