Tracking Forecast Error Type, Frequency and Magnitude with the Forecast Error Package Page 45 - Developments in Business Simulation and Experiential Learning, volume 38, 2011 ABSTRACT The Online Forecast Error Package is used to identify and track the type, frequency and magnitude of sales forecast errors over the course of competition. Competing partici- pant teams use this package to determine the extent of end- ing inventory, stockouts, and overtime production for each of their nine strategic business units at the end of each de- cision period. Later, the teams use the package to present consolidated sales forecast errors during the team presen- tation at the end of competition. Then, the teams present a sales forecast using a linear unrestricted single equation multiple regression model and check its accuracy. The use of the Forecast Error Package highlights the need for and significance of forecast accuracy in improving team per- formance. INTRODUCTION The Forecast Error Package enables competing partici- pant teams in the marketing simulation COMPETE (Faria, 2006) to assess the accuracy of their sales forecasts for each of their nine strategic business units (SBUs). SBUs are specific product offerings in specific regions that have specific target markets with specific needs and purchase motivations, a specific set of strategies, facing a specific set of competitors with specific competing strategies. The Forecast Error Package is used in conjunction with the Shipment Analysis worksheet. Based on the demand fore- cast, ending inventory and a safety stock, the Shipment Analysis worksheet enables participant teams to decide on the level of shipments. Later, the Forecast Error Package enables them to assess the congruence between supply based on shipments and demand based on the marketing mix. The new Excel-based Forecast Error Package auto- matically extracts relevant data via external links from the Excel-version of the COMPETE simulation results. The Excel-version of the simulation results are generated by the instructor/administrator from the original dos-text based COMPETE simulation results. Later, the Excel-version of the simulation results are uploaded to the COMPETE Online Decision Entry System (CODES) repository for subsequent access by competing participant teams. Only relevant data on ending inventory, stockouts and overtime production in units are extracted in order to determine the congruence between demand created via marketing mix decisions and supply based on shipment decisions. Demand forecasts are needed to determine the potential profitability of a marketing strategy. Accurate demand forecasts enable marketing managers to allocate available resources effec- tively and efficiently. This decision support package saves substantial time needed to identify and enter the relevant data and reduces the potential for data entry error. DECISION SUPPORT SYSTEMS Several scholars have commented on the value of in- cluding decision support software/systems in computer simulations (Keys & Biggs, 1990; Teach, 1990; Gold & Pray, 1990; Wolfe & Gregg, 1989). In addition, the litera- ture is replete with references to the use and impact of deci- sion support systems with computer simulations (Affisco & Chanin, 1989, 1990; Burns & Bush 1991; Cannon et al., 1993; Fritzsche et al., 1987; Grove et al., 1986; Halpin, 2006; Honaiser & Sauaia 2006; Markulis & Strang 1985; Mitri et al., 1998; Muhs & Callen, 1984; Nulsen et al., 1994; Palia, 1989, 1991; Peach, 1996; Schellenberger, 1983; Shane & Bailes, 1986; Sherrell et al., 1986; Wingen- der & Wurster, 1987; Woodruff, 1992). Decision support systems (DSSs) are defined as …a collection of data, systems, tools, and techniques with sup- porting software and hardware by which an organization gathers and interprets relevant information from business and environment and turns it into a basis for…action (Little, 1979; Burns & Bush, 1991). In addition, they are defined as computer-based information systems that sup- port the process of structuring problems, evaluating alterna- tives, and selecting actions for more effective management (Forgionne, 1988). Further, they are described as the hard- ware and software that permit decision-makers to deal with a specific set of related problems by providing tools that amplify a manager‟s judgment (Sprague, 1980). DSSs used with business simulations yield several benefits. These include greater depth of understanding of simulation activity with resulting increase in planning (Keys et al., 1986), in-depth understanding of quantitative techniques as students visualize the results of their applica- tions, sensitivity to weaknesses in techniques used, and experience in capitalizing on their strengths (Fritzsche et al., 1987). Other benefits include minimization of paper- work and errors, error-free graphical representation of out- put, a competitive tool with increasing value as simulation progresses, and potential for participants to create their own DSSs (Burns & Bush, 1991). In addition, DSSs enhance TRACKING FORECAST ERROR TYPE, FREQUENCY AND MAGNITUDE WITH THE FORECAST ERROR PACKAGE Aspy P. Palia University of Hawaii at Manoa aspy@hawaii.edu mailto:aspy@hawaii.edu Page 46 - Developments in Business Simulation and Experiential Learning, volume 38, 2011 understanding of complex business relationships and pro- vide additional value over time (Halpin, 2006). Further, DSSs provide realism, relevance, literacy, flexibility and opportunity for refinement (Sherrell et al., 1986). Some authors contend that combining an active student generated database in the form of a simulation game with a DSS will result in improved decision making, lead to im- proved pro-active rather than re-active strategic planning, and result in improved simulation game performance and enhanced learning (Muhs & Callen, 1984). Others have reported no support for the premise that DSS usage im- proves small group decision making effectiveness (Affisco & Chanin, 1989), and that DSS usage to support manufac- turing function decisions resulted in decreased manufactur- ing costs and increased “earnings/cost of goods sold” ratio in the second year of play (Affisco & Chanin, 1990). Given the inconsistent findings with regard to the effi- cacy of DSSs reported in the literature, does DSS usage increase decision effectiveness and/or enhance learning? One scholar notes that while the DSS assists the decision maker, it does not make decisions, nor can it substitute for intelligent analysis and synthesis (Schellenberger, 1983). In addition, as with other computer-based or experiential learning techniques, the effectiveness of DSSs or the deci- sions made are less important than the insights they gener- ate. The level of insight generated depends heavily on the clear explanation of the purpose, significance, assumptions, usage, and limitations of the DSS and underlying concepts applied, by the instructor. In addition, the level of insight generated depends heavily on the debriefing process used by the instructor to crystallize student learning (Cannon et al., 1993). The primary purpose of this paper is to present this new user-centered learning tool that helps to prepare stu- dents for sales forecasting, production scheduling, inven- tory management, and marketing decision-making respon- sibilities in their future careers. The objective of this deci- sion support package is to provide participant teams the opportunity to forecast sales, schedule shipments, and man- age inventory effectively and efficiently. SALES FORECASTING A major responsibility of marketing is the preparation of sales forecasts. First, market opportunities are identified through marketing research. Then, the size, growth and profitability of each market opportunity are measured and/ or forecasted. Sales forecasts are used (a) by finance to raise the needed cash for investment and operations, (b) by manufacturing to establish capacity and output levels, (c) by purchasing to acquire the necessary supplies, and (d) by human resources to hire the needed workers (Kotler, 2003). Accurate sales forecasts facilitate effective and effi- cient allocation of scarce resources. Over-estimates of de- mand lead to several problems. First, excess inventory uses up valuable shelf space and leads to obsolescence. Next, scarce working capital blocked up in inventory carrying charges [funds used or borrowed (a) by manufacturers to produce goods, or (b) by retailers to purchase goods] can- not be used for other purposes such as R&D or promotional expenses. Third, storage charges are incurred to store ex- cess inventory in public or private warehouses. Finally, margins are reduced when excess inventory is removed through end-of-year clearance sales. Under-estimates of demand lead to a different set of problems. First, stock-outs lead to wasted shelf space. Next, insufficient inventory leads to lost sales and conse- quent lost margins. Third, failure to keep up with customer demand may necessitate the use of limited and expensive overtime production leading to lower profitability. Finally, and most importantly, the firm may lose customers, when prospects facing an empty store shelf, try an alternative brand or go to an alternative store, and are satisfied by the competitive offering. Given the detrimental impact of inac- curate forecasts, marketers use a variety of sales forecasting techniques in order to forecast sales accurately (Palia, 2004). Several scholars have referred to the use of sales fore- casting by business simulation participants (Arellano & Hopkins, 1992; Burns, 1977; Cannon & Burns, 1999; Can- non & Ternan, 1997; Cosenza et al., 1985; Day, 1986; Dickinson & Faria, 1994; Dickinson et al., 1990; Edwards, 1987; Faria & Whiteley, 1990; Faria & Wellington, 2004; Faria & Wellington, 2005; Goosen, 2008; Gray et al., 1986; Green & Faria, 1995; Hall, 1987; Hemasi et al., 1987; Keys, 1977, Keys & Gooding, 1990; Keys, 1982; Keys, 1987; Napier, 1974; Neuhauser, 1976; Peach, 1996; Peach & Platt, 2001; Schellenberger, 1981; Schellenberger & Masters, 1986; Snow, 1976; Sord, 1977; Sugges, 1982; Thavilkulwat, 1986; Wellington & Faria, 2006; Wellington et al., 2008; Wellington et al., 2009; Wellington et al., 2010; Whiteley, 1989). SALES FORECASTING TECHNIQUES Marketers forecast sales by (a) extending past behav- ior, and/or by (b) predicting future behavior. Extending past behavior techniques are applied when past sales data are available. These techniques tend to be more quantita- tive. They extend past data into the future, and assume that the future will be like the past. These techniques include trend extension, the factor method (using one or more fac- tors such as the Buying Power Index, and the SIC code to forecast the sales of industrial products), time series analy- sis, the use of leading series, and indices such as the con- sumer price index, producer price index, and the index of leading economic indicators (McCarthy & Perreault, 1987). Predicting future behavior techniques do not rely on the availability of past data. Instead, they rely on judg- ment, and are used when there are changing conditions in the marketplace or changes in the marketing mix used. They are also used to forecast the sale of unstable (fashion) Page 47 - Developments in Business Simulation and Experiential Learning, volume 38, 2011 goods and new products. These techniques include the Jury of Executive Opinion, salespeople‟s estimates, surveys of final buyers, retailers and/or wholesalers, panels of stores and/or final consumers, market tests of existing prod- ucts, test markets of new products, the substitute method, and needs analysis (Perreault & McCarthy, 1996). There is no best method of forecasting in all circumstances. Confi- dence in the accuracy of sales forecasts is derived by cor- roborating the results using two or more methods (McCarthy & Perreault, 1984). FORECAST ACCURACY/ERROR Forecasting accuracy has been proposed and/or used to assess management performance in business simulation games with mixed results (Anderson & Lawton, 1990; de Souza et al., 2010; Gosenpud et al., 1984; Hand & Sims 1975; Newgren et al., 1981; Peach & Platt, 2001; Teach, 1989; Teach, 2006; Teach, 2007; Washbush, 2003; Washbush & Gosen, 2002). Hand and Sims (1975) used path analysis to investigate the relationships among thirteen performance criteria (Swanson, 1977). They were able to reduce the number of performance criteria from thirteen to two – sales forecast- ing error (the primary “driving variable”) and profits (the primary “end result”). Gosenpud et al. (1984) used multi- ple regression to ascertain the influence of elven independ- ent variables on organizational effectiveness, and per- formed factor analysis to determine the relationship among the independent variables. They found that forecast accu- racy with the largest regression coefficient (Beta = 0.32) affected ROE significantly. Teach (1989) investigated the relationship between forecasting accuracy and simulation performance. He concluded that market share, unit sales, net cash flow and profit/loss forecast accuracy were di- rectly related to simulation performance. Further, his ongo- ing research (Teach 1987, 1989, 1990, 1993a, 1993b, 2007) suggests that there is a positive correlation between fore- cast accuracy and simulation performance. Washbush & Gosen (2002) examined the relationship between learning and forecast accuracy but did not find a consistent correlation between the two variables. They suggest that forecast accuracy may be a proxy for other simulation performance measures. In a follow-up study, Washbush (2003) evaluated the proposition that there is a correlation between forecast accuracy and total enterprise simulation performance. The major findings of this study were consistent with Teach‟s findings that forecast accu- racy correlates with simulation performance. Most re- cently, deSouza, Bernard & Cannon (2010) used multiple regression to evaluate whether reduction in forecast error can be used as a predictor of team performance. They found that forecast accuracy explained 40.75% of the vari- ance in company performance, and that forecast accuracy for high-level functions (general management) had the greatest predictive impact and low-level functions (sales, human resources, and finance) the lowest. In addition, sev- eral scholars have discussed forecasting techniques (Harmon et al., 2009; Kamath & Roy, 2005; Wei & Wang, 2007), forecast accuracy (Dong & Zhu, 2008; Makatosoris & Chang, 2008; Piirainen et al., 2009) and forecast error (Ahlert & Block, 2010) in industry. Given the relationship between forecast accuracy and simulation performance, an effort is needed to learn and use a variety of forecasting techniques. As there is no sin- gle best forecasting approach for all situations, an under- standing of the strengths and weaknesses of each technique needs to be understood. Next the sales forecasts need to be monitored in order to establish the frequency of stockouts, overtime and excessive ending inventory and the conse- quences thereof to team performance and customer satis- faction. Once the consequences of forecast error on reve- nues, costs, margins, and customer perceptions are clearly understood, simulation participants will better understand the need to learn and develop forecasting skills. They will be motivated to learn the ability to develop accurate sales forecasts in order to improve subsequent team perform- ance. The Forecast Error Package tracks the type, fre- quency, and magnitude of forecast error for each of their nine SBUs over the course of competition. THE MARKETING SIMULATION COM- PETE COMPETE (Faria, 2006) is a marketing simulation designed to provide students with marketing strategy devel- opment and decision-making experience. Competing stu- dent teams are placed in a complex, dynamic, and uncertain environment. The participants experience the excitement and uncertainty of competitive events and are motivated to be active seekers of knowledge. They learn the need for and usefulness of mastering an underlying set of decision- making principles. Competing student teams plan, implement, and control a marketing program for three high-tech products in three regions Region 1 (R1), Region 2 (R2) and Region 3 (R3) within the United States. These three products are a Total Spectrum Television (TST), a Computerized DVD/Video Editor (CVE) and a Safe Shot Laser (SSL). The features and benefits of each product and the characteristics of con- sumers in each region are described in the student manual. Based on a marketing opportunity analysis, a mission state- ment is generated, specific and measurable company goals are set, and marketing strategies are formulated to achieve these goals. Constant monitoring and analysis of their own and competitive performance helps the teams better under- stand their markets and improve their decisions. Each decision period (quarter), the competing teams make a total of 74 marketing decisions with regard to mar- keting their three brands in the three regional markets. These decisions include nine pricing decisions, nine ship- ment decisions, three sales force size decisions, nine sales force time allocation decisions, one sales force salary deci- Page 48 - Developments in Business Simulation and Experiential Learning, volume 38, 2011 sion, one sales force commission decision, twenty-seven advertising media decisions, nine advertising content deci- sions, three quality-improvement R&D decisions, and three cost-reduction R&D decisions. Successful planning, im- plementation, and control of their respective marketing programs require that each company constantly monitor trends in its own and competitive decision variables and resulting performance. COMPETE ONLINE DECISION ENTRY SYSTEM (CODES) The COMPETE Online Decision Entry System (CODES) is a web-based simulation interface that enables competing participant teams with Internet access, to regis- ter their teams, enter and submit their decisions, and subse- quently to retrieve and print out their results from a remote site (Palia, Mak & Roussos, 2000). The teams log in to the CODES website (Palia & Mak, 2001; Palia, Mak & Roussos, 2000). Their login is vali- dated against a database of participating teams for each industry, and they have access to their decisions and print- outs (results) for all prior decision periods. Once the team ID and password are validated against a database of participating teams, the user (participant) is presented with a personalized Welcome screen with several options. In addition to the “Main Menu” option, the user is presented with one or more of three dynamic links “Grades,” “Handouts,” and “Performance” only if and when the corresponding files are uploaded to their industry folder on the web server by the administrator (Palia, 2006; Palia, 2007; Palia, 2008). At the “Main Menu” webpage they select “Enter Deci- sions” to enter their team decisions prior to the decision deadline. At the decision deadline, the administrator downloads the team decision files, runs the simulation, and uploads the text and Excel versions of the simulation re- sults to the Web Server. Later, the teams log in to CODES, proceed to the Main Menu, and select “View Results” to view their team performance results in either text or Excel format. The competing participant teams are provided with access to online strategic market planning (Palia, 1995; Palia, 1996; Palia et al., 2002; Palia, 2010), positioning (Palia, 1997; Palia et al., 2003), sales forecast model- building (Palia, 2004), budgeting (Palia, 2007), market testing (Palia & Roussos, 2006), target profit pricing (Palia, 2008), strategic business unit analysis (Palia, 2009) and other performance enhancing tools (Palia, 2005) in order to facilitate user-centered learning (Palia et al., 2000). FORECAST ERROR FLOW PACKAGE The web-based Forecast Error Package Version 2.0 is accessible online to competing participant teams in the marketing simulation COMPETE. The Forecast Error Package Version 2.0 is a zipped folder “Forecast Error.zip” which consists of an Excel workbook “Forecast Er- ror.xlsx” (with external links to each of x.xls COMPETE output files) and x.xls Excel version of sample COMPETE Forecast Error – Overtime Production Worksheet (in Units) Figure 1 Period ==> 1 2 3 4 5 6 7 8 9 10 11 12 TST - 1 0 0 0 0 0 0 0 0 0 0 0 0 TST - 2 0 0 0 0 0 0 0 0 0 0 0 0 TST - 3 0 0 0 0 0 0 0 0 0 0 0 0 CVE - 1 0 0 0 141 146 0 0 0 0 0 0 0 CVE - 2 0 0 0 2,420 558 0 0 1,198 0 0 0 1,314 CVE - 3 0 0 0 1,150 0 0 0 649 0 0 0 0 SSL - 1 0 0 1,986 0 0 0 0 0 0 0 0 0 SSL - 2 0 0 4,939 0 0 0 2,299 0 0 465 0 0 SSL - 3 0 0 0 0 0 0 0 0 0 0 0 0 Page 49 - Developments in Business Simulation and Experiential Learning, volume 38, 2011 output for all specified periods “x”. This Forecast Er- ror.xlsx workbook consists of three worksheets (a) Over- time Production worksheet, (c) Ending Inventory work- sheet and (c) Consolidated Forecast Error worksheet. The Overtime Production worksheet (see Figure 1) consists of external links to the quarterly COMPETE out- put files. This worksheet extracts the overtime production / shipments in units for each product in each region during each decision period from the quarterly results for the first twelve quarters (three years) of operation. A multiple bar chart graphically presents the overtime production / ship- ments for all nine SBUs in all twelve decision periods. Zero units of overtime production / shipments could either signify ending inventory (where shipments exceed demand) or instances where overtime production is not triggered. Overtime production in COMPETE is only triggered when 50 percent of the current stockout or 20 percent of the cur- rent shipping request, whichever is less, exceeds 100 units. While the initial stockout condition results in lost sales and potential lost customers, the overtime production, when triggered results in higher unit production costs. These unit overtime production costs are 15 percent greater than unit production costs during regular production hours. Conse- quently, the combination of stockouts and overtime produc- tion signify lower (lost) sales due to stockouts and in- creased unit overtime production costs which result in lower margins. The Ending Inventory worksheet (see Figure 2) con- sists of external links to the quarterly COMPETE output files. This worksheet extracts the ending inventory in units for each product in each region during each decision period from quarterly results for the first twelve quarters (three years) of operation. A multiple bar chart graphically pre- sents the ending inventory for all 9 SBUs in all 12 decision periods. Conditional formatting is used to signify instances when there is excess inventory with resulting high inven- tory carrying costs, storage charges, and potentially costly markdowns or clearance sales. Excess inventory is deemed to exist when there is an ending inventory of more than 100 units of the relatively high-priced TST, more than 500 units of the medium-priced CVE, and more than 1000 units of the relatively low-priced SSL. The Forecast Error Package extracts relevant data via external links on overtime production / shipments and end- ing inventory for each SBU during each period from the Excel version of the COMPETE simulation results. First, this package extracts overtime production from the “Overtime Production / Shipments” table on the respective “Quality, Cost, OT, Shipments” worksheet for each deci- sion period (quarter). Next, this package extracts ending inventory from the “Shipments and Inventory By Region By Product” table on the respective “Quality, Cost, OT, Shipments” worksheet for each decision period. The relevant data are extracted from the COMPETE Results Excel workbook x.xls to the Forecast Error work- book as indicated in the Data Extraction Table (see Figure 3). The Excel worksheet (tab), page number in the Excel- version of the COMPETE results printout, and cell refer- ences for each account are shown in the COMPETE Re- sults Workbook table (on the right). The corresponding cell references for each account are shown in the Forecast Error workbook table (on the left). For instance, the Over- Forecast Error – Ending Inventory Worksheet (in Units) Figure 2 Period ==> 1 2 3 4 5 6 7 8 9 10 11 12 TST - 1 400 0 242 758 528 192 333 750 562 187 281 522 TST - 2 0 0 110 585 778 0 226 638 467 170 356 378 TST - 3 142 0 2 538 501 132 113 487 0 74 403 66 CVE - 1 0 5,450 4,103 0 0 4,937 6,820 1,536 1,331 835 3,363 995 CVE - 2 715 3,556 3,985 0 0 6,183 237 0 378 3,349 4,354 0 CVE - 3 11 4,524 3,939 0 0 3,075 3,442 0 3,959 4,303 2,630 370 SSL - 1 53,361 6,405 0 36,538 17,703 15,700 9,222 34,813 5,176 19,272 12,423 20,524 SSL - 2 34,648 633 0 19,913 12,431 11,662 0 30,968 12,303 0 10,855 22,041 SSL - 3 23,745 4,492 3,801 18,411 6,898 17,693 0 30,398 2,817 4,941 4,916 17,409 Page 50 - Developments in Business Simulation and Experiential Learning, volume 38, 2011 time Production for TST in Region 1 (TST – 1) in cell B4 on the “Forecast Error – Overtime Production” worksheet in Figure 1 is extracted from cell G18 in the “Overtime Production / Shipments” table on the “Quality, Cost, OT, Shipments” worksheet of the COMPETE results workbook. Similarly, the Ending Inventory for CVE in Region 2 (CVE – 2) in cell B8 on the “Forecast Error – Ending Inventory” worksheet in Figure 2 is extracted from cell I28 in the “Shipments and Inventory By Region By Product” table on the “Quality, Cost, OT, Shipments” worksheet of the COMPETE results workbook. The Consolidated Forecast Error worksheet (see Figure 4) consolidates, color-codes and presents forecast errors due to stockouts, overtime and ending inventory for all products in all regions during all 12 decision periods. In addition, the worksheet presents the number and percentage of instances of stockouts, overtime, excessive ending in- ventory, and reasonably accurate forecasts for all nine Data Extraction Table Figure 3 Account Cell Ref. Worksheet (Tab) Page # Account Cell Ref. Overtime Production Worksheet (Tab) Period 1 TST in Region 1 (TST - 1) B4 from ==> Quality, Cost, OT, Shipments 7 TST - 1 Overtime Production G18 TST in Region 2 (TST - 2) B5 from ==> Quality, Cost, OT, Shipments 7 TST - 2 Overtime Production G19 TST in Region 3 (TST - 3) B6 from ==> Quality, Cost, OT, Shipments 7 TST - 3 Overtime Production G20 CVE in Region 1 (CVE - 1) B7 from ==> Quality, Cost, OT, Shipments 7 CVE - 1 Overtime Production H18 CVE in Region 2 (CVE - 2) B8 from ==> Quality, Cost, OT, Shipments 7 CVE - 2 Overtime Production H19 CVE in Region 3 (CVE - 3) B9 from ==> Quality, Cost, OT, Shipments 7 CVE - 3 Overtime Production H20 SSL in Region 1 (SSL - 1) B10 from ==> Quality, Cost, OT, Shipments 7 SSL - 1 Overtime Production I18 SSL in Region 2 (SSL - 2) B11 from ==> Quality, Cost, OT, Shipments 7 SSL - 2 Overtime Production I19 SSL in Region 3 (SSL - 3) B12 from ==> Quality, Cost, OT, Shipments 7 SSL - 3 Overtime Production I20 Ending Inventory Worksheet (Tab) TST in Region 1 (TST - 1) B4 from ==> Quality, Cost, OT, Shipments 7 TST - 1 Ending Inventory F27 TST in Region 2 (TST - 2) B5 from ==> Quality, Cost, OT, Shipments 7 TST - 2 Ending Inventory F28 TST in Region 3 (TST - 3) B6 from ==> Quality, Cost, OT, Shipments 7 TST - 3 Ending Inventory F29 CVE in Region 1 (CVE - 1) B7 from ==> Quality, Cost, OT, Shipments 7 CVE - 1 Ending Inventory I27 CVE in Region 2 (CVE - 2) B8 from ==> Quality, Cost, OT, Shipments 7 CVE - 2 Ending Inventory I28 CVE in Region 3 (CVE - 3) B9 from ==> Quality, Cost, OT, Shipments 7 CVE - 3 Ending Inventory I29 SSL in Region 1 (SSL - 1) B10 from ==> Quality, Cost, OT, Shipments 7 SSL - 1 Ending Inventory L27 SSL in Region 2 (SSL - 2) B11 from ==> Quality, Cost, OT, Shipments 7 SSL - 2 Ending Inventory L28 SSL in Region 3 (SSL - 3) B12 from ==> Quality, Cost, OT, Shipments 7 SSL - 3 Ending Inventory L29 Data Extraction from COMPETE Results Workbook.xls To Forecast Error Workbook COMPETE Forecast Error Workbook COMPETE Results Workbook x.xls (x = Period Number) Period ==> 1 2 3 4 5 6 7 8 9 10 11 12 TST - 1 400 0 242 758 528 192 333 750 562 187 281 522 TST - 2 0 0 110 585 778 0 226 638 467 170 356 378 TST - 3 142 0 2 538 501 132 113 487 0 74 403 66 CVE - 1 0 5,450 4,103 141 146 4,937 6,820 1,536 1,331 835 3,363 995 CVE - 2 715 3,556 3,985 2,420 558 6,183 237 1,198 378 3,349 4,354 1,314 CVE - 3 11 4,524 3,939 1,150 0 3,075 3,442 649 3,959 4,303 2,630 370 SSL - 1 53,361 6,405 1,986 36,538 17,703 15,700 9,222 34,813 5,176 19,272 12,423 20,524 SSL - 2 34,648 633 4,939 19,913 12,431 11,662 2,299 30,968 12,303 465 10,855 22,041 SSL - 3 23,745 4,492 3,801 18,411 6,898 17,693 0 30,398 2,817 4,941 4,916 17,409 Stockouts 9 instances 8.33% Bare shelves, lost sales, and lost customers Overtime 12 instances 11.11% Bare shelves, lost sales, lost customers, & overtime costs Excessive Ending Inventory 79 instances 73.15% Inventory carrying cost, storage charge, & clearance sales Reasonably Accurate Forecasts 8 instances 7.41% only Increased sales, lower costs & higher margins Total Number of Instances 108 Legend: Excess Inventory Condition: Stockouts SBU Overtime TST >100 units Excessive Ending Inventory CVE >500 units Reasonably Accurate Forecasts SSL >1000 units Instances Percent ImplicationsCondition Consolidated Forecast Error Worksheet (in Units) Figure 4 Page 51 - Developments in Business Simulation and Experiential Learning, volume 38, 2011 SBUs over all twelve periods. Conditional formatting is used to highlight instances of stockouts (in green color), instances of overtime production (in brown color), in- stances of excessive ending inventory (in yellow color), and instances of reasonably accurate forecasts (in white color) for specific products in specific regions during each of the twelve decision periods. Conditional counts are used to determine the number and percentage of instances of stockouts (when both overtime production and ending in- ventory are zero), overtime production, ending inventory, and reasonably accurate forecasts. The instances of rea- sonably accurate forecasts are the remaining instances cal- culated by subtracting the sum of instances of stockouts, ending inventory and overtime from a total of 108 instances (12 decision periods x 9 SBUs). No data entry is required. The use of external links ensures relevant data are ex- tracted from relevant sources (statements) in the simulation results and precludes data entry error. Cell comments clar- ify variables extracted from the COMPETE results to both the Overtime Production and Ending Inventory worksheets (see Figures 5 & 6). In addition, cell comments clarify functions used to identify, count, and color code each type of forecast error in the Consolidated Forecast Error work- sheet (see Figure 7). Period ==> 1 2 3 4 5 6 7 8 9 10 11 12 TST - 1 0 0 0 0 0 0 0 0 0 0 0 0 TST - 2 0 0 0 0 0 0 0 0 0 0 0 0 TST - 3 0 0 0 0 0 0 0 0 0 0 0 0 CVE - 1 0 0 0 141 146 0 0 0 0 0 0 0 CVE - 2 0 0 0 2,420 558 0 0 1,198 0 0 0 1,314 CVE - 3 0 0 0 1,150 0 0 0 649 0 0 0 0 SSL - 1 0 0 1,986 0 0 0 0 0 0 0 0 0 SSL - 2 0 0 4,939 0 0 0 2,299 0 0 465 0 0 SSL - 3 0 0 0 0 0 0 0 0 0 0 0 0 TST-1 Overtime Production extracted from Overtime Production / Shipments Table for Period 1. CVE-1 Overtime Production extracted from Overtime Production / Shipments Table for Period 5. SSL-1 Overtime Production extracted from Overtime Production / Shipments Table for Period 9. Forecast Error – Overtime Production Worksheet With Cell Comments (in Units) Figure 5 Period ==> 1 2 3 4 5 6 7 8 9 10 11 12 TST - 1 400 0 242 758 528 192 333 750 562 187 281 522 TST - 2 0 0 110 585 778 0 226 638 467 170 356 378 TST - 3 142 0 2 538 501 132 113 487 0 74 403 66 CVE - 1 0 5,450 4,103 0 0 4,937 6,820 1,536 1,331 835 3,363 995 CVE - 2 715 3,556 3,985 0 0 6,183 237 0 378 3,349 4,354 0 CVE - 3 11 4,524 3,939 0 0 3,075 3,442 0 3,959 4,303 2,630 370 SSL - 1 53,361 6,405 0 36,538 17,703 15,700 9,222 34,813 5,176 19,272 12,423 20,524 SSL - 2 34,648 633 0 19,913 12,431 11,662 0 30,968 12,303 0 10,855 22,041 SSL - 3 23,745 4,492 3,801 18,411 6,898 17,693 0 30,398 2,817 4,941 4,916 17,409 TST-1 Ending Inventory extracted from 'Shipments and Inventory By Region By Product' Table in Period 1. CVE-1 Ending Inventory extracted from 'Shipments and Inventory By Region By Product' Table in Period 5. SSL-1 Ending Inventory extracted from 'Shipments and Inventory By Region By Product' Table in Period 9. Forecast Error – Ending Inventory Worksheet With Cell Comments (in Units) Figure 6 Period ==> 1 2 3 4 5 6 7 8 9 10 11 12 TST - 1 400 0 242 758 528 192 333 750 562 187 281 522 TST - 2 0 0 110 585 778 0 226 638 467 170 356 378 TST - 3 142 0 2 538 501 132 113 487 0 74 403 66 CVE - 1 0 5,450 4,103 141 146 4,937 6,820 1,536 1,331 835 3,363 995 CVE - 2 715 3,556 3,985 2,420 558 6,183 237 1,198 378 3,349 4,354 1,314 CVE - 3 11 4,524 3,939 1,150 0 3,075 3,442 649 3,959 4,303 2,630 370 SSL - 1 53,361 6,405 1,986 36,538 17,703 15,700 9,222 34,813 5,176 19,272 12,423 20,524 SSL - 2 34,648 633 4,939 19,913 12,431 11,662 2,299 30,968 12,303 465 10,855 22,041 SSL - 3 23,745 4,492 3,801 18,411 6,898 17,693 0 30,398 2,817 4,941 4,916 17,409 Stockouts 9 instances 8.33% Bare shelves, lost sales, and lost customers Overtime 12 instances 11.11% Bare shelves, lost sales, lost customers, & overtime costs Excessive Ending Inventory 79 instances 73.15% Inventory carrying cost, storage charge, & clearance sales Reasonably Accurate Forecasts 8 instances 7.41% only Increased sales, lower costs & higher margins Total Number of Instances 108 Legend: Excess Inventory Condition: Stockouts SBU Overtime TST >100 units Excessive Ending Inventory CVE >500 units Reasonably Accurate Forecasts SSL >1000 units Instances Percent ImplicationsCondition =SUM('Overtime Production'!B4,'Ending Inventory'!B4) =SUM('Overtime Production'!B4,'Ending Inventory'!B4) =SUM('Overtime Production'!B4,'Ending Inventory'!B4) =COUNTIFS(Overtime,"=0", Ending_Inventory,"=0") =COUNTIF(TST,">100") +COUNTIF(CVE,">500") +COUNTIF(SSL,">1000") Consolidated Forecast Error Worksheet With Cell Comments(in Units) Figure 7 Page 52 - Developments in Business Simulation and Experiential Learning, volume 38, 2011 FORECAST ERROR PACKAGE PROCESS First, the participant teams download and unzip the Forecast Error.zip folder. Next, they login to CODES and download, rename and save the Excel version of results for all twelve periods (quarters) “x” in the unzipped “C:\Forecast Error” directory. Then, they update the Fore- cast Error.xls workbook with team data. For instance, to update the Forecast Error worksheet with team data, they first open the unzipped Forecast Error folder, then open the Forecast Error.xls workbook, and finally click “Update file” in the pop-up menu that appears. The overtime units are extracted from the Overtime Production / Shipments table on the Quality, Cost, OT, Shipments page of the Excel version of the COMPETE results for the first twelve periods (quarters) of operation. The ending inventory in units is extracted from the Ship- ments and Inventory By Region By Product table on the Quality, Cost, OT, Shipments page of Excel version of the COMPETE results for the twelve periods (quarters) of op- eration. The Consolidated Forecast Error worksheet consoli- dates, color-codes, and counts the number of instances of stockouts, overtime, excessive ending inventory (defined as 100 or more units of the relatively high-priced TST, 500 or more units of the relatively average-priced CVE, and 1000 or more units of the relatively low-priced SSL), and the remaining instances of reasonably accurate forecasts. Based on the extracted data, the Forecast Error Pack- age calculates the percentage of instances of stockouts, overtime, excessive ending inventory and reasonably accu- rate forecasts during the twelve periods of competition. The use of external links ensures relevant data are extracted from relevant sources (statements) in the simulation results and precludes data entry error. Cell comments (see Figure 5,6 & 7) clarify variables, cell formulae and functions used. Color-coded cells specify where data are extracted and dif- ferentiate among instances of stockout, overtime, excessive ending inventory and reasonably accurate forecasts. STRENGTHS AND LIMITATIONS The Online Forecast Error Package is used to track and present the type, frequency and magnitude of forecast error during the twelve periods (quarters) of competition. The consolidated forecast error matrix consolidates, color- codes, and counts the number of instances of stockouts, overtime, excessive ending inventory and reasonably accu- rate forecasts during the twelve periods of competition. This matrix is presented at the end of the insightful and revealing “Errors Made and Lessons Learned” section of the team presentation at the end of the semester. The con- solidated forecast error matrix demonstrates the pervasive nature of forecast error. In addition, this matrix highlights the need for more accurate forecasts in order to reduce end- ing inventory, inventory carrying charges, storage charges, and clearance sales when forecasts are overly optimistic, and the need to avoid stockouts, lost sales, costly overtime production and lost customers when forecasts are overly pessimistic. Finally, the consolidated forecast error matrix provides a smooth transition to the use of a linear unre- stricted single-equation multiple regression model to more accurately forecast sales. Positive anecdotal student feedback was received at the end of the Spring and Fall 2010 semesters. Some stu- dents reported that the decision support packages were very useful and helpful. They indicated that the automatic ex- traction feature saved a lot of time that would otherwise be necessary to identify, enter and compute the necessary fig- ures. They hoped that it would continue to be used in the future as it definitely made a difference. Other students indicated that they did not make full use of the DSS. Admittedly, sales forecasting is a complex and criti- cally important task that requires considerable effort, judg- ment and experience. The user needs to select and use suit- able sales forecasting techniques that are either quantitative and objective or subjective and judgmental depending upon the availability of past data, changes in the market and mar- keting mix, and relative stability of product life cycles. Next, the user needs to corroborate the results in order to derive confidence in the forecast. Then the user needs to determine the level of shipments based on the level of end- ing inventory, the sales forecast and the level of safety stock. In order to improve forecast accuracy, the user needs to track the type, frequency, and magnitude of fore- cast error, understand the underlying reasons, and take cor- rective action. Despite these limitations, the Forecast Error Package is a simple yet powerful web-based user-centered learning tool that extracts relevant data from the simulation results, precludes data entry error, and saves considerable time involved in identifying and entering relevant data. Yet, in order to maximize learning about sales forecasting, and actualize the potential of Forecast Error Package, the in- structor needs to (a) explain the purpose, significance, as- sumptions, usage, and limitations of this DSS package as well as sales forecast model building, (b) require inclusion of a sample analysis and subsequent sales forecast in a team report and/or presentation, and (c) test students on their understanding of the underlying concepts at the end of the semester. CONCLUSION The Online Forecast Error Package is a user-centered learning tool that helps to prepare students for sales fore- casting and marketing decision-making responsibilities in their future careers. This package enables users to apply sales forecasting, monitor the type, frequency and magni- tude of forecast error, and thereby assess the accuracy of their forecasts and their impact on team performance. For- mer researchers have identified sales forecasting error (the Page 53 - Developments in Business Simulation and Experiential Learning, volume 38, 2011 primary “driving variable”) and profits (the primary “end result” as important simulation performance criteria (Hand & Sims, 1975). They have concluded that forecast accu- racy affects ROE significantly (Gosenpud et al, 1984)., and that there is a positive correlation between forecast accu- racy and simulation performance (Teach, 1987, 1989, 1990, 1993a, 1993b, 2007). Other researchers have found no consistent relationship between learning and forecast accu- racy (Washbush & Gosen, 2002). More recently, scholars have determined that forecast accuracy correlates with simulation performance (Washbush, 2003), and that fore- cast accuracy explained 40.75% of the variance in company performance (deSouza, Bernard & Cannon, 2010). Given the importance of forecast accuracy to simula- tion performance, participants can use the Online Forecast Error Package to (a) determine the type, frequency and magnitude of forecast error for each of their nine SBUs over the 12 periods (quarters) of competition, (b) better understand the significance of accurate forecasting, (c) learn different forecasting techniques, and (d) improve their forecasting skills. Further, this online Forecast Error Pack- age may be used to empirically test the relationships among sales forecasting, simulation performance and enhanced learning. In addition, the package facilitates the integration of computers, the Internet and the World Wide Web into the marketing curriculum. REFERENCES Affisco, J. F., & Chanin, M. N. (1989). The Impact Of Decision Support Systems On The Effectiveness Of Small Group Decisions – An Exploratory Study. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Exercises, 16, 132-135. (Reprinted from Bernie Keys Library (11th ed.)) Affisco, J. F., & Chanin, M. N. (1990). The Impact of De- cision Support Systems On The Effectiveness of Small Group Decisions – Revisited. In J.A. Smith (Ed.), Developments in Business Simulation and Experiential Learning, 17, 1-5. (Reprinted from Bernie Keys Li- brary (11th ed.)) Ahlert, K. H., & Block, C. (2010). Assessing the Impact of Price Forecast Errors on the Economics of Distributed Storage Systems. In R.H. Sprague, Jr. (Ed.), Proceed- ings of the 43rdAnnual Hawaii International Confer- ence on System Sciences, 43, 1-10. (Reprint from IEEE Computer Society) Anderson, P. H., & Lawton, L. (1990). The Relationship Between Financial Performance and Other Measures Of Learning On A Simulation Exercise. In J.A. Smith (Ed.), Developments in Business Simulation & Experi- ential Exercises, 17, 6-10. (Reprinted from Bernie Keys Library (11th ed.)) Arellano, F. E., & Hopkins, W. E. (1992). Modelling Eco- nomic Environments In Business Simulations: Some Comparisons and Recommendations. In J.A. Smith (Ed.), Developments in Business Simulation & Experi- ential Exercises, 19, 7-10. (Reprinted from Bernie Keys Library (11th ed.)) Burns, A. C. (1977). A Simplified, Noncomputerized Mar- keting Channels Game. In New Horizons in Simula- tion Games and Experiential Learning, 4, 301-308. (Reprinted from Bernie Keys Library (11th ed.)) Burns, A. C., & Bush, R. F. (1991). Using DIS „n DAT as a Decision Support System for a Marketing Simulation Game. In J.A. Smith (Ed.), Developments in Business Simulation and Experiential Learning, 18, 5-10. (Reprinted from Bernie Keys Library (11th ed.)) Cannon, H. M., & Burns, A. C. (1999). A Framework For Assessing The Competencies Reflected In Simulation Performance. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Exercises, 26, 40- 44. (Reprinted from Bernie Keys Library (11th ed.)) Cannon, H. M., & Ternan, A. (1997). Contextually- Anchored Business Simulations. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Learning, 24, 322-328. (Reprinted from Bernie Keys Library (11th ed.)) Cannon, H. M., Carroll, P. G., & Seamons, B. L. (1993). Using The IdeaFisher Idea Generation System as a Decision Support System in Marketing Strategy Courses. In J.A. Smith (Ed.), Developments in Busi- ness Simulation and Experiential Learning, 20, 27-30. (Reprinted from Bernie Keys Library (11th ed.)) Cosenza, R. M., Boone, L. E., & Kurtz, D. L. (1985). Inte- grating Microcomputers In The Marketing Curriculum Through The Use of Marketing Compuprobs. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Exercises, 12, 54-55. (Reprinted from Bernie Keys Library (11th ed.)) Day, R. L. (1986). A Sales Management Simulation For the PC: An Integrative Tool For Sales Management Courses. In J.A. Smith (Ed.), Developments in Busi- ness Simulation & Experiential Exercises, 13, 63-65. (Reprinted from Bernie Keys Library (11th ed.)) De Souza, M. P., Bernard, R.S., & Cannon, H. M. (2010). Another Look At The Use Of Forecasting Accuracy On The Assessment Of Management Performance In Business Simulation Games. In J.A. Smith (Ed.), De- velopments in Business Simulation & Experiential Learning, 37, 68-78. (Reprinted from Bernie Keys Library (11th ed.)) Dickinson, J. R., & Faria, A.J. (1994). A Random-Strategy Criterion For Validity Of Simulation Game Participa- tion. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Exercises, 21, 35-39. (Reprinted from Bernie Keys Library (11th ed.)) Dickinson, J. R., Whiteley, T. R., & Faria, A. J. (1990). An Empirical Investigation of The Internal Validity of A Marketing Simulation Game. In J.A. Smith (Ed.), De- velopments in Business Simulation & Experiential Ex- ercises, 17, 47-52. (Reprinted from Bernie Keys Li- brary (11th ed.)) Page 54 - Developments in Business Simulation and Experiential Learning, volume 38, 2011 Dong, S., & Zhu, K. (2008). The Business Value of CRM Systems: A Resource-Based Perspective. In R. H. Sprague, Jr. (Ed.), Proceedings of the 41st Hawaii In- ternational Conference on System Sciences, Vol. 41, 1- 10. (Reprint from IEEE Computer Society) Edwards, W. F. (1987). Learning Macroeconomic Theory and Policy Analysis via Microcomputer Simulation. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Exercises, 14, 50-53. (Reprinted from Bernie Keys Library (11th ed.)) Faria, A. J., (2006). COMPETE: A Dynamic Marketing Simulation, 5th ed. Windsor, CA: University of Win- dsor. Faria, A. J., Nulsen, Jr., R. O., & Roussos, D. S. (1994). COMPETE:A Dynamic Marketing Simulation, (4th ed.). Burr Ridge, IL: Irwin. Faria, A. J., & Whiteley, T. R. (1990). An Empirical Evaluation of The Pedagogical Value of Playing A Simulation Game In A Principles of Marketing Course. In J.A. Smith (Ed.), Developments in Business Simula- tion & Experiential Exercises, 13, 204-206. (Reprinted from Bernie Keys Library (11th ed.)) Faria, A. J., & Wellington, W. J. (2004). Validating Busi- ness Simulations: Does High Market Share Lead To High Profitability? In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Learning, 31, 332-336. (Reprinted from Bernie Keys Library (11th ed.)) Faria, A. J., & Wellington, W. J. (2005). Validating Busi- ness Simulations: Does High Product Quality Lead To High Profitability? In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Learning, 32, 117-121. (Reprinted from Bernie Keys Library (11th ed.)) Forgionne, G. A. (1988), Building Effective Decision Sup- port Systems. Business, 38 (1), 19-30. Fritzche, D. J., Rodich, G. W., & Cotter, R. V. (1987). Integrating Decision Support Systems and Business Games. In J.A. Smith (Ed.), Developments in Business Simulation and Experiential Learning, 14, 63-66. (Reprinted from Bernie Keys Library (11th ed.)) Fritzche, D. J., Rodich, G. W., & Cotter, R. V. (1987). Integrating Decision Support Systems and Business Games. In J.A. Smith (Ed.), Developments in Business Simulation and Experiential Learning, 14, 63-66. (Reprinted from Bernie Keys Library (11th ed.)) Gold, S. C., & Pray, T. F. (1990). Modeling Demand in Computerizing Business Simulations. Guide to Busi- ness Gaming and Experiential Learning, Chapter 8. Association for Business Simulation and Experiential Learning (ABSEL), 117-138. East Brunswick/Kogan Page, London: Nichols/GP Publishing. Goosen, K. R. (2008). Do Price Strategies Work in Busi- ness Simulations? In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Learning, 35, 273-288. (Reprinted from Bernie Keys Library (11th ed.)) Gosenpud, J., Miessing, P., & Milton, C. J. (1984). A Re- search Study On Strategic Decisions In A Business Simulation. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Learning, 11, 161- 165. (Reprinted from Bernie Keys Library (11th ed.)) Gray, G., Andersen, A., Gentry, J. W., & Manzer, L. L. (1986). Teaching Reevaluation of Salespersons Through The Use of A Simulation Game. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Exercises, 13, 66-69. (Reprinted from Bernie Keys Library (11th ed.)) Green, D., & Faria, A. J. (1995). Are Good Strategies Con- sistently Good? In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Exercises, 22, 31- 37. (Reprinted from Bernie Keys Library (11th ed.)) Grove, S. J., Pickett, G. H., & Williams, R. H. (1986). The Subjective Side of The Decision Support System A Pitfall For The Panacea. In J.A. Smith (Ed.), Develop- ments in Business Simulation and Experiential Learn- ing, 13, 170-173. (Reprinted from Bernie Keys Li- brary (11th ed.)) Hall, D.R. (1987). Developing Various Student Learning Abilities via Writing, the Stock Market Game, and Modified Marketplace Game. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Exercises, 14, 84-87. (Reprinted from Bernie Keys Library (11th ed.)) Halpin, A. L. (2006). A Decision Support System For Planning Sales, Production, and Plant Addition With Manager: A Computer Simulation. In J.A. Smith (Ed.), Developments in Business Simulation and Expe- riential Learning, 33, 289-293. (Reprinted from Bernie Keys Library (11th ed.)) Hand, H. H., & Sims, H. P. (1975). Statistical Evaluation of Complex Gaming Performance. Management Sci- ence, 21, 6 (February), 708-717. Harmon, R., Demirkan, H., Hefley, B., and Auseklis, N. (2009). Pricing Strategies for Information Technology Services: A Value-Based Approach. In R.H. Sprague, Jr. (Ed.), Proceedings of the 42nd Annual Hawaii Inter- national Conference on System Sciences, 42, 1-10. (Reprint from IEEE Computer Society) Hemasi, M., Graf, L. A., & Kellogg, C. E. (1987). A Com- parison of Performance Attitudes And Behaviors of MBA and BBA Students In A Simulation Environ- ment: A Preliminary Investigation. In J.A. Smith (Ed.), Developments in Business Simulation & Experi- ential Exercises, 14, 90-94. (Reprinted from Bernie Keys Library (11th ed.)) Honaiser, E., & Sauaia, A. C. A. (2006). Decision Support System For Demand Forecasting in Business Games, In J.A. Smith (Ed.), Developments in Business Simula- tion and Experiential Learning, 33, 223-231. (Reprinted from Bernie Keys Library (11th ed.)) Kamath B., Narasimha, & Roy, R. (2005). Supply Chain Structure Design for a Short Lifecycle Product: A Loop Dominance Based Analysis. In R.H. Sprague, Jr. Page 55 - Developments in Business Simulation and Experiential Learning, volume 38, 2011 (Ed.), Proceedings of the 38th Annual Hawaii Interna- tional Conference on System Sciences, 38, 1-10. (Reprint from IEEE Computer Society) Keys, B. (1977). Review of Learning Research In Business Gaming. In J.A. Smith (Ed.), Computer Simulation and Learning Theory, 3, 173-184. (Reprinted from Bernie Keys Library (11th ed.)) Keys, B., & Gooding, C. (1990). Introducing Executive MBA Programs With Management Games. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Exercises, 17, 199. (Reprinted from Bernie Keys Library (11th ed.)) Keys, J. B. (1982). The Delivery, Administration, and Evaluation of An Executive Development Program Using A Total Enterprise Business Game. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Exercises, 9, 162-163. (Reprinted from Bernie Keys Library (11th ed.)) Keys, J. B. (1987). Total Enterprise Business Games: An Evaluation. Developments in Business Simulation & Experiential Exercises, 14, 104-108. (Reprinted from Bernie Keys Library (11th ed.)) Keys, J. B. & Biggs, W. B. (1990). A Review of Business Games. In J.W. Gentry (Ed.), Guide to Business Gam- ing and Experiential Learning, Chapter 5. Association for Business Simulation and Experiential Learning (ABSEL), 48-73. East Brunswick/Kogan Page, Lon- don: Nichols/GP Publishing. Keys, J. B., Burns, O. M., Case, T. L., & Wells, R. A. (1986). Performance and Attitudinal Affects of a De- cision Support Package in a Business Game. In J.A. Smith (Ed.), Developments in Business Simulation and Experiential Learning, 13, 221-226. (Reprinted from Bernie Keys Library (11th ed.)) Kotler, P. (2003). Marketing Management, 11th ed. Upper Saddle River, NJ: Prentice-Hall. Lai, Y. P. and Siau, T. L. (2003). EBiz Game: A Scalable Online Business Simulation Game For Entrepreneur- ship Training. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Learning, 30, 270- 278. (Reprinted from Bernie Keys Library (11th ed.)) Little, J. D. C. (1979). Decision Support Systems for Mar- keting Managers, Journal of Marketing, 43 (Summer), 9-26. Makatosoris, C., & Chang, Y. S. (2008). Beyond CRM: A System to Bridge the Gap between the customer and the Global Manufacturing Supply Chain. In R.H. Sprague, Jr. (Ed.), Proceedings of the 41st Annual Ha- waii International Conference on System Sciences, 41, 1-10. (Reprint from IEEE Computer Society) Markulis, P. M., & Strang, D. R. (1985). The Use of Deci- sion Support Systems (DSS) and Operations Research/ Management Science (OR/MS) Techniques to En- hance the Learning Experience of Students Participat- ing in Computerized Simulations, In J.A. Smith (Ed.), Developments in Business Simulation and Experiential Learning, 12, 30-34. (Reprinted from Bernie Keys Library (11th ed.)) McCarthy, E. J., & Perreault, Jr., W. D. (1984) Basic Mar- keting, 8th ed. Homewood, IL: Irwin. McCarthy, E. J., & Perreault, Jr., W. D. (1987). Basic Marketing, 9th ed. Homewood, IL: Irwin. Mitri, M., Karimalis, G., Cannon, H., & Yaprak, A. (1998). The Market Access Planning System (MAPS): Com- puter-Based Decision Support System For Facilitating Experiential Learning in International Business, In J.A. Smith (Ed.), Developments in Business Simula- tion and Experiential Learning, 25, 101-107. (Reprinted from Bernie Keys Library (11th ed.)) Muhs, W. F., & Callen, R. W. (1984). Incorporating Deci- sion Support Systems Into Management Simulation Games: A Model and Methodology, In J.A. Smith (Ed.), Developments in Business Simulation and Expe- riential Learning, 11, 261-266. (Reprinted from Bernie Keys Library (11th ed.)) Napier, H. S. (1974). Autocratic vs Democratic Decision Making: The Executive Game As An Experiment. In J.A. Smith (Ed.), Simulations, Games and Experiential Learning Techniques, 1, 291-294. (Reprinted from Bernie Keys Library (11th ed.)) Neuhauser, J.J. (1976). Business Games Have Failed. Academy of Management Review, 1, 124-129. Nulsen, R. O., Jr., Faria, A. J., & Roussos, D. S. (1994). The Use of Decision Support Systems With a Market- ing Simulation: The Future is Now. In J.A. Smith (Ed.), Developments in Business Simulation and Expe- riential Exercises, 21, 169. (Reprinted from Bernie Keys Library (11th ed.)) Newgren, K. E., Stair, R. M., & Kuehn, R. R. (1981). De- cision Efficiency and Effectiveness In A Business Simulation. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Exercises, 8, 263- 265. (Reprinted from Bernie Keys Library (11th ed.)) Palia, A. P. (2010). Checking Financial Balance of Target Brand Portfolio With The Strategic Market Plan Cash Flow Package. In J.A. Smith (Ed.), Developments in Business Simulation and Experiential Exercises, 37, 157-69. (Reprinted from Bernie Keys Library (11th ed.)) Palia, A. P. (1995). Comparative Static Analysis With The COMPETE PPA Package: A Strategic Market Plan- ning Tool. In J.A. Smith (Ed.), Developments in Busi- ness Simulation and Experiential Exercises, 22, 130-1. (Reprinted from Bernie Keys Library (11th ed.)) Palia, A. P. (2007). Online Budgeting and Marketing Con- trol With the Proforma Analysis Package. In J.A. Smith (Ed.), Developments in Business Simulation and Experiential Learning, 34, 149-54. (Reprinted from Bernie Keys Library (11th ed.)) Palia, A. P. (2005). Online Cumulative Simulation Team Performance Package. In J.A. Smith (Ed.), Develop- ments in Business Simulation and Experiential Learn- Page 56 - Developments in Business Simulation and Experiential Learning, volume 38, 2011 ing, 32, 233-39. (Reprinted from Bernie Keys Library (11th ed.)) Palia, A. P. (2009). Online Marketing Control With The Strategic Business Unit Analysis Package. In J.A. Smith (Ed.), Developments in Business Simulation and Experiential Exercises, Vol. 36, 91-101. (Reprinted from Bernie Keys Library (11th ed.)) Palia, A. P. (2004). Online Sales Forecasting With the Multiple Regression Analysis Data Matrices Package. In J.A. Smith (Ed.), Developments in Business Simula- tion and Experiential Learning, 31, 53-7. (Reprinted from Bernie Keys Library (11th ed.)) Palia, A. P. (1997). Plotting Brand Trajectories With The COMPETE PPM Package: A Market Segmentation Analysis and Positioning Tool. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Learning, 24, 172-176. (Reprinted from Bernie Keys Library (11th ed.)) Palia, A. P. (1996). Strategic Analysis of the Product Port- folio With The COMPETE PPA Package: A Strategic Market Planning Tool. In J.A. Smith (Ed.), Develop- ments in Business Simulation and Experiential Exer- cises, 23, 93. (Reprinted from Bernie Keys Library (11th ed.)) Palia, A. P. (1989). Sensitivity Analysis With The COM- PETE IFPS/Personal Student Analysis Package: A Marketing Decision Support System. In J.A. Smith (Ed.), Developments in Business Simulation and Expe- riential Exercises, 16, 141-4. (Reprinted from Bernie Keys Library (11th ed.)) Palia, A. P. (1991). Strategic Market Planning With The COMPETE Product Portfolio Analysis Package: A Marketing Decision Support System. In J.A. Smith (Ed.), Developments in Business Simulation and Expe- riential Exercises, 18, 80-3. (Reprinted from Bernie Keys Library (11th ed.)) Palia, A. P. (2008). Target Profit Pricing With the Web- based Breakeven Analysis Package. In J.A. Smith (Ed.), Developments in Business Simulation and Expe- riential Learning, 35, 197-204. (Reprinted from Bernie Keys Library (11th ed.)) Palia, A. P., & Roussos, D. S. (2006). Online Market Test Laboratory With the MINISIM Program. In J.A. Smith (Ed.), Developments in Business Simulation and Experiential Learning, 33, 238-41. (Reprinted from Bernie Keys Library (11th ed.)) Palia, A. P., De Ryck, J. & Mak, W.K. (2003). Interactive Online Positioning With the Web-based Product Posi- tioning Map Graphics Package. In J.A. Smith (Ed.), Developments in Business Simulation and Experiential Learning, 30, 202-6. (Reprinted from Bernie Keys Library (11th ed.)) Palia, A. P., De Ryck, J. & Mak, W. K. (2002). Interactive Online Strategic Market Planning With the Web-based Boston Consulting Group (BCG) Matrix Graphics Package. In J.A. Smith (Ed.), Developments in Busi- ness Simulation and Experiential Learning, 29, 140-2. (Reprinted from Bernie Keys Library (11th ed.)) Palia, A. P., & Mak, W. K. (2001). An Online Evaluation of The COMPETE Online Decision Entry System (CODES). In J.A. Smith (Ed.), Developments in Busi- ness Simulation and Experiential Learning, 28, 188- 91. (Reprinted from Bernie Keys Library (11th ed.)) Palia, A. P., Mak, W. K., & Roussos, D. S. (2000). Facili- tating Learning in the New Millennium With The COMPETE Online Decision Entry System (CODES). In J.A. Smith (Ed.), Developments in Business Simula- tion and Experiential Learning, 27, 250-1. (Reprinted from Bernie Keys Library (11th ed.)) Peach, E. B. (1996). Enhancing Simulation Learning Through Objectives And Decision Support Systems. In J.A. Smith (Ed.), Developments in Business Simula- tion & Experiential Exercises, 23, 61-67. (Reprinted from Bernie Keys Library (11th ed.)) Peach, E. B., & Platt, R. G. (2001). Gilding The Lily, En- hancing The Pedagogical Experience, Or Fixing What Isn‟t Broke: The Dilemma of New Simulation Ver- sions. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Learning, 28, 196-200. (Reprinted from Bernie Keys Library (11th ed.)) Perreault, W. D., Jr., & McCarthy, E. J. (1996). Basic Marketing: A Global-Managerial Approach. Chicago, IL: Irwin. Piirainen, K., Elfvengren, K., Korpela, J., and Tuominen, M. (2009). Improving the effectiveness of business process development through collaboration engineer- ing: a method for process elicitation. In R.H. Sprague, Jr. (Ed.), Proceedings of the 42nd Annual Hawaii Inter- national Conference on System Sciences, Vol. 42, 1- 10. (Reprint from IEEE Computer Society) Schellenberger, R. E. (1981). Critical Variables In Re- search On The Educational Value of Management Games. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Exercises, 8, 70. (Reprinted from Bernie Keys Library (11th ed.)) Schellenberger, R. E. (1983). MANSYM III Decision Sup- port System Demonstration. In J.A. Smith (Ed.), De- velopments in Business Simulation and Experiential Learning, 10, 69-71. (Reprinted from Bernie Keys Library (11th ed.)) Schellenberger, R. E., & Masters, L. A. (1986). An Exam- ple: The Use of Management Games On Microcom- puters By Computer Novices. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Exercises, 13, 204-206. (Reprinted from Bernie Keys Library (11th ed.)) Shane, B., & Bailes, J. (1986). A Decision Support System For Capital Funds Forecasting. In J.A. Smith (Ed.), Developments in Business Simulation and Experiential Learning, 13, 216-220. (Reprinted from Bernie Keys Library (11th ed.)) Sherrell, D., Russ, K. R., & Burns, A. C. (1986). Enhanc- ing Mainframe Simulations via Microcomputers: De- Page 57 - Developments in Business Simulation and Experiential Learning, volume 38, 2011 signing Decision Support Systems. In J.A. Smith (Ed.), Developments in Business Simulation and Expe- riential Learning, 13, 207-211. (Reprinted from Bernie Keys Library (11th ed.)) Snow, C.C. (1976) A Comment on Business Policy Teach- ing Research. Academy of Management Review, 1, 133-135. Sord, B. H. (1977). Comments About Learning Experi- ences By Participants In A General Management Business Simulation Conducted At The Univer- sity of Texas at Austin. In J.A. Smith (Ed.), Com- puter Simulations and Learning Theory, 3, 258-270. (Reprinted from Bernie Keys Library (11th ed.)) Sprague, R. H., Jr. (1980). A Framework for the Develop- ment of Decision Support Systems. Management In- formation Systems Quarterly, 4 (December), 1-26. Sugges, Jr., P. R. (1982). An Evaluation of SLIM (A Sys- tem Laboratory For Information Management). In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Exercises, 9, 38-39. (Reprinted from Bernie Keys Library (11th ed.)) Swanson, L. A. (1977). Evaluation of Performance In Managerial Simulations: A Managerial Coefficients Model. In J.A. Smith (Ed.), Computer Simulations and Learning Theory, 3, 372-377. (Reprinted from Bernie Keys Library (11th ed.)) Teach, R. D. (1990). Designing Business Simulations. In J.W. Gentry (Ed.), Guide to Business Gaming and Ex- periential Learning, Chapter 7, Association for Busi- ness Simulation and Experiential Learning (ABSEL), pp. 93-116. East Brunswick/Kogan Page, London: Nichols/GP Publishing. Teach, R. D, (1993a). Forecasting Accuracy as a Perform- ance Measure in Business Simulations. Simulation & Gaming, 24:4 (December), 476-490. Teach, R. D. (1993b). Forecasting and Management Abil- ity: A Response to Wolfe. Simulation & Gaming, 24:1 (March), 63-72. Teach, R. D. (1987). Profit: The False Prophet. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Learning, 14, 205-207. (Reprinted from Bernie Keys Library (11th ed.)) Teach, R. D. (1990). Profit: The False Prophet in Business Gaming. Simulation & Gaming, 21:1 (March), 12-26. Teach, R. (2007). Forecasting Accuracy and Learning: A Key To Measuring Business Game Performance. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Learning, 34, 57-66. (Reprinted from Bernie Keys Library (11th ed.)) Teach, R. (2006). Forecasting Accuracy and Learning: The Key To Measuring Simulation Performance. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Learning, 33, 48-57. (Reprinted from Bernie Keys Library (11th ed.)) Teach, R. (1989). Using Forecasting Accuracy As A Measure of Success In Business Simulations. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Exercises, 16, 103-107. (Reprinted from Bernie Keys Library (11th ed.)) Thavikulwat, P. (1986). Another Chance: A Multi- Leveled, Macro Business Game For Microcomputers. In J.A. Smith (Ed.), Developments in Business Simula- tion & Experiential Exercises, 13, 143-145. (Reprinted from Bernie Keys Library (11th ed.)) Washbush, J. B. (2003). Simulation Performance and Forecast Accuracy – Is That All? Developments in Business Simulation & Experiential Learning, 30, 250- 253. (Reprinted from Bernie Keys Library (11th ed.)) Washbush, J., & Gosen, J. (2002). Total Enterprise Simu- lation Learning Compared To Traditional Learning In The Business Policy Course. In J.A. Smith (Ed.), De- velopments in Business Simulation & Experiential Learning, 29, 281-286. (Reprinted from Bernie Keys Library (11th ed.)) Wei, H. L., & Wang, E. T. G. (2007). Creating Strategic Value from Supply Chain Visibility – the Dynamic Capabilities View. In R.H. Sprague, Jr. (Ed.), Pro- ceedings of the 40th Annual Hawaii International Con- ference on System Sciences, 40, 1-10. (Reprint from IEEE Computer Society) Wellington, W., & Faria, A. J. (2006). Validating Business Simulations: Do Simulations Exhibit Natural Market Structures. In J.A. Smith (Ed.), Developments in Busi- ness Simulation & Experiential Learning, 33, 118-123. (Reprinted from Bernie Keys Library (11th ed.)) Wellington, W. J., Faria, A. J., & Hutchinson, D. (2008). Are Good Strategy Decisions Consistently Good? A Real-Time Investigation. In J.A. Smith (Ed.), Devel- opments in Business Simulation & Experiential Learn- ing, 35, 340-346. (Reprinted from Bernie Keys Li- brary (11th ed.)) Wellington, W. J., Hutchinson, D., & Faria, A. J. (2009). Marketing Simulation Game Decision Making Experi- ence And Its Impact On Indecisiveness Among Intro- ductory Marketing Students. In J.A. Smith (Ed.), De- velopments in Business Simulation & Experiential Learning, 36, 223-230. (Reprinted from Bernie Keys Library (11th ed.)) Wellington, W. J., Hutchinson, D., & Faria, A. J. (2010). The Impact Of Playing A Marketing Simulation Game On Perceived Decision Making Ability Among Intro- ductory Marketing Students. In J.A. Smith (Ed.), De- velopments in Business Simulation & Experiential Learning, 37, 23-32. (Reprinted from Bernie Keys Library (11th ed.)) Whiteley, T. R., & Faria, A. J. (1989). A Study of The Relationship Between Student Final Exam Perform- ance And Simulation Game Participation. In J.A. Smith (Ed.), Developments in Business Simulation & Experiential Exercises, 16, 78-82. (Reprinted from Bernie Keys Library (11th ed.)) Wingender, J., & Wurster, J. (1987). Oil And Gas Well Investment Analysis Using The Lotus 1-2-3 Decision Support System, In J.A. Smith (Ed.), Developments in Page 58 - Developments in Business Simulation and Experiential Learning, volume 38, 2011 Business Simulation and Experiential Learning, 14, 245-249. (Reprinted from Bernie Keys Library (11th ed.)) Wolfe, J., & Gregg, J. (1989). On the Efficacy of Manage- rial Decision Support Systems in a Business Gaming Environment, Proceedings of the International Simu- lation and Gaming Association, 102-109. Woodruff, C. K. (1992). A Graphics Application Exten- sion For A Simulated Decision Support System Envi- ronment, In J.A. Smith (Ed.), Developments in Busi- ness Simulation and Experiential Learning, 18, 5-10. (Reprinted from Bernie Keys Library (11th ed.)) Table of Contents Volume 38, 2011 Simulated Tabletop Exercise for Risk Management - Anti Bio-terrorism Scenario Simulated Tabletop Exercise From Business Games to Simulations - Simuworlds & Microworlds Demand Equation Redux: The Design and Functionality of the Gold/Pray Model in Computerized Business Simulations Managing Client-Based Learning: Insights from Successful Teaching Project Courses in Marketing Tracking Forecast Error Type, Frequency and Magnitude with the Forecast Error Package Responding to Facilitate Collaboration Simulating Sudden Change and the Value of Timely Information Managing Human Resources Simulation A Study on Collectivism and Group Decision-Making: An International Comparison of Japan, China, and Russia Using a Gaming Simulation The Use of Management Games in the Management Research Agenda Gaming On-Line: A Simulation Application Positioning and Performance in Simulated Networks Supply Chain Management: A Simulation Application Simulation as a Teaching Method in Strategic Management Distance Studies Entrepreneurship: A Game of Risk and Reward Phase II: The Start-Up Return to the Paradise Islands: From Confrontation to Cooperation Effect on Market Performance of Displaying Supply and Demand Curves in a Business Simulation Appreciating Complexity: The Chief of Staff of the Army Game Managing Organizations: Experiential MBA Course Teaches Alternatives to the Machine Model A Simulation Model for Analyzing the Night-Time Emergency Health Care System in Japan The Continuing Evoluation of Assessing Project Management as an Academic Learning Outcome (ALO) Should College Instructors Change Their Teaching Styles to Meet the Millenial Student? The Mouse Game and its Effects on Team Interdependence Learning from the Gulf Oil Spill to Prepare for a Brighter Future: A New Game Engaging Stake Holders in Triple Bottom Line Accounting & Strategic Planning Video Killed the Biblio Star: The Impact of Digital Media on Student Learning Outcomes Exploring Motivation: Using Emoticons to Map Student Motivation in a Business Game Exercise An Alternative to PC and Internet Based Simulations: The Internet Integrated Mode MiddleState University -- A Crisis in Education Complexity Avoidance, Narcissism and Experiential Learning Examining the Cognitive, Affective, and Psychomotor Dimensions in Management Skill Development Through Experiential Learning: Developing a Framework A Situational Leadership Exercise Based on the Biology of a Starfish JOGAI CEFET -- The Industrial Administration Undergraduate Game Would You Take a Marketing Man to a Quick Service Restaurant? Modeling Corporate Social Responsibility In A Food Service Menu-Management Simulation Use of a Simulation in a Large Class Environment for a Marketing Principles Class: A Qualitative Analysis of Whether Learning Objectives were Met A Team Based Information Literacy Exercise ABSEL Marketing Communications Plan An Interdisciplinary Study of the Impact of Playing a Marketing Simulation Game on Student Knowledge of Management Accounting/Finance Principles Analyzing Construction Planning of Interiro Finish Work of Apartment Building by Simulation Doing Murder One Again The Simple Business Game and Simulation Transfering the Knowledge of Middle Management to Novices Infectious Disease Simulation Model for Estimation of Spreading Understanding the Relative Influence of Several Factors in ERP Simulation Performance: An Exploration of Ecological Validity Tragedy of the Commons: An Exercise Using Clickers to Illustrate and Teach a Key Concept in Negotiations The Meaning of Firm Demand in Business Simulations If the Games Work, Why Aren't More Faculty Willing to Play? Those Who Do and Those That Don't: A Study of Engaged and Disengaged Business Game Players