A MISUSE OF PIMS FOR THE VALIDATION OF MARKETING MANAGEMENT SIMULATION GAMES Developments in Business Simulation and Experiential Learning, Volume 30, 2003 A MISUSE OF PIMS FOR THE VALIDATION OF MARKETING MANAGEMENT SIMULATION GAMES John R. Dickinson University of Windsor bjd@uwindsor.ca ABSTRACT The results of two recent studies that, respectively, put forth and seemingly replicate a PIMS-based criterion for simulation game validity portend a dramatic shift in the design of competitive marketing management games. The two studies suggest that competitors’ strategies may largely be ignored in the formulation of a company’s marketing strategy. However, as conceived and as applied in the two studies, the criterion is itself invalid. This paper challenges the conceptualization of the proposed validation criterion, the design of the studies based on it, and the implication for marketing strategy formulation. INTRODUCTION AND BACKGROUND In the tradition of validating simulation games against real-world criteria (Naylor et al. 1966, p. 40; Napier & House 1990; House, Parks, & Lindstrom 1994; House & Taylor 1992), Green & Faria (1995, hereafter G&F) introduced an ostensible finding of the Profit Impact of Marketing Strategies (PIMS) project (Buzzell & Gale 1987), specifically the PIMS finding, “...that 80 percent of the variance in a company’s performance can be explained by its environment” (G&F, p. 34). In other words, “...successful strategies in a particular marketplace/economic environment will continue to be successful in similar environments-even if competition is changed” (G&F, p. 34). Subsequently, Neal (1999, hereafter Neal) purported to confirm “...the results of G&F in that the strategies continued to be successful in an environment with different competitors” (p. 118). However, (1) the operationalization and empirical findings of PIMS expressly contradict the premise of the G&F and Neal studies, (2) a huge quantity and diversity of other empirical research contradicts that premise, and (3) the common research design used by G&F and Neal does not support the conclusion drawn by the researchers, i.e., the design is invalid. Business games may be classified as competitive or noncompetitive (Biggs 1990, p. 25), synonymously interactive or non-interactive (Meurs & Choffray 1975), respectively, or, in perhaps less ambiguous terminology, dependent-across- firms or independent-across-firms (Thavikulwat 1988), respectively. “Thus, games in which the demand [and other outcomes] for individual firms depends on the decisions of the other firms in the game would be >dependent-across-firms’ while those in which demand for individual firms is not dependent on the decisions of other firms in the game would be >independent-across-firms” (Biggs 1990, pp. 25-26). Consider, then, a simulation game condition in which the marketing strategies of a company’ competitors change, the marketing strategy of the focal company remains unchanged, and the effectiveness of the focal company’ strategy is unaffected. Within the taxonomy of business games just described, the game could not be classified as competitive or dependent-across-firms; the game would be classified as noncompetitive or independent-across-firms. POSITING OF A PIMS-BASED CRITERION FOR VALIDITY AND A REPLICATION Green and Faria (1995) derived their criterion from a conclusion they attribute to Schoeffler (1993). Ostensibly, A...80 percent of the variance in a company’ performance can be explained by its environment” (G&F, p. 34). G&F interpret Schoeffler’ finding to mean that the effectiveness of a marketing strategy is largely independent of competitors’ strategies. In other words, “...successful strategies in a particular marketplace/ economic environment will continue to be successful in similar environments-even if competition is changed” (G&F, p. 34). Subsequently, Neal (1999, hereafter Neal) purported to confirm “...the results of G&F in that the strategies continued to be successful in an environment with different competitors” (p. 118). G&F demonstrated the application of their proposed criterion using data from a Compete (Faria, Nulsen, and Roussos 1994) simulation game competition involving 25 industries, each industry comprising five companies. The leading company in each industry, as measured by cumulative earnings, at the end of the competition was identified. Then, in a one-time manipulation, the 25 leading teams were randomly reassigned to different industries-i.e., the leading company in one industry replaced the leading company in a second industry-and the “competition” was rerun using the strategy decisions of the companies in the reconfigured industries. Eighteen of the 25 originally leading companies (72%) again had the highest cumulative earnings in the industries to which they had been reassigned and an additional three companies (12%) had the second highest cumulative earnings. On these results, G&F concluded that, “...a winning strategy remains a sound strategy even when transferred to a new competitive environment” (p. 34). Subsequently, Neal replicated and extended G&F’ study 99 mailto:bjd@uwindsor.ca Developments in Business Simulation and Experiential Learning, Volume 30, 2003 “...to explore the principle that one can be successful even when competitor actions are ignored totally” (Neal, p. 120). Using the Markstrat2 (Larr�ch� and Gatignon 1990) game, Neal “...confirmed the results of Green and Faria in that the strategies continued to be successful in an environment with different competitors” (p. 118). Neal, though, extended G&F’ study in three material ways. First, in addition to examining rank position of the reassigned companies, he also examined actual earnings amount. Second, instead of a single, one-time reassignment of industry-leading companies, he implemented all possible reassignments among four industries, a total of 12 reassignments. Third, he imposed a budget constraint on companies, the amount of the budget being a function of company profits earned, whereas G&F did not impose a budget constraint. His finding most directly comparable to that of G&F was that six (50%) of the 12 reassigned companies finished first and six (50%) finished second in the industries to which they were reassigned. THE PIMS DATA BASE Contrary to the interpretation by G&F, with regard to marketing strategy the PIMS data base contains relatively few measures that themselves are not dependent on competitors. Table 1 presents a summary of competitor-independent and relative-to-competitors PIMS measures. There is relatively little information in the PIMS data base that would provide a basis for concluding that the effectiveness of marketing strategy is anything but dependent on competitor strategies. TABLE 1: PIMS Marketing Mix Measures Competitor-Independent Expenditures Strategy Relative to Competitors product/service R & D ($) sales force ($) advertising and sales promotion including media ($) advertising media only ($) new products (percent of sales) vertical integration (value added as percent of sales) other marketing expenses ($) product quality sales force expenditures sales promotion expenditures advertising expenditures new products (percent of sales) backward vertical integration forward vertical integration product/company image prices breadth of product line quality of services Sources: Abell and Hammond (1979, pp. 291-320); Buzzell and Gale (1997, Appendix A) In drawing principles from the abundant empirical research based on the PIMS project, the foremost is that, “In the long run, the most important single factor affecting a business unit’ performance is the quality of its products and services, relative to those of competitors” (Buzzell and Gale 1987, p. 7, italics added). “Not only do management’ own policies and programs affect sales, but so do those of competitors. A realistic approach to explaining market share change must, therefore, include all of the important elements of marketing strategy and must somehow relate any single competitor’ actions to the contending moves of rivals” (Buzzell and Wiersma 1981, p. 136). All or nearly all of the PIMS-based researches relating business performance, usually either in terms of return on investment or market share, to marketing strategy have used the relative-to-competitors strategy measures. Across at least 36 different PIMS-based studies (list available upon request from the author), significant relationships have been found for every marketing strategy variable defined relative to competitors. There is near-universal empirical evidence within the PIMS project that the success of a marketing strategy is dependent on the strategies of competitors. THE PIMS PRINCIPLE: 80% INCLUDES, NOT EXCLUDES, COMPETITOR STRATEGIES The sole reference cited between G&F and Neal in support of their validity criterion is Schoeffler (1993). While the reference is literally correct, it is interpreted in those studies contrary to its correct meaning. Schoeffler himself makes clear the correct meaning: “When we try to understand the variance between [very profitable and very unprofitable businesses], the laws of the marketplace account for up to 80 percent of that variance. This means that the characteristics of the served market, of the business itself, and of its competitors constitute about 80 percent of the reasons for success or failure...” (Schoeffler 1983, p. 23-4, italics added). Schoeffler’ “marketplace” does not exclude competitors, as interpreted by G&F and Neal; the marketplace expressly includes competitors. Schoeffler (1983) attributes the remaining 20 percent of variance to “...the operating skill or luck of the management...”(p. 23-4). The 80% figure derives from a multiple regression 100 Developments in Business Simulation and Experiential Learning, Volume 30, 2003 analysis specifying return on investment as the dependent variable and 37 independent variables plus numerous interaction terms; the 80% figure is the resulting R2 (Schoeffler 1977; 1983). Among the key explanatory variables listed by Schoeffler (1977) are product quality, vertical integration, and price; all marketing strategy variables and all measured relative to competitors. The very model yielding the oft-cited 80% figure itself includes competitor effects in the definitions of numerous key explaining variables. CORROBORATION OF COMPETITOR EFFECTS Substantive areas of marketing theory and empirical study other than PIMS also do not support the competitor-independence criterion. Briefly: • Virtually every modern strategic marketing text (e.g., Aaker 1998, Chapter 4; Czinkota & Kotabe 2001, Chapter 3; Jain 2000, Chapter 4; Kotler, Cunningham, and Turner 2001, Chapter 8; Stauble 2000, Chapter 6) integrates competitors into strategy formulation. • In consumer behavior theory the prominent role of competitors is described by Hawkins, Best, and Coney (2001, p. 15): “It is not possible to consistently do a better job of meeting customer needs than the competition without a thorough understanding of the competition’ capabilities and strengths.” • Sethuraman, Srinivasan, and Kim (1999) identified 1,060 published empirical estimates of price cross- elasticity. About 85 percent of the estimates were positive. • Grewal et al.’ (1997) meta-analysis of 77 studies that compared comparative advertising. Across 12 criteria for advertising effectiveness, nine significant (α=.05) differences were found between the two types of advertising. That is, the presence of a competitor’ strategy, i.e., its brand, alters the effectiveness of a marketing (advertising) strategy. • Hanssens, Parsons & Schultz (2001, p. 321) present a general model for own- and cross-elasticity for any marketing strategy variable”...in which the marketing decision variable is expressed in relative [to competitors] share form...” Parsons and Schultz (1976, pp. 140-143) reviewed 28 econometric sales response models, fifteen of which incorporated competitive effects. • In the 1994 Survey of Marketing Research (Kinnear and Root 1995) for the subsample of consumer companies-the subsample most similar to the types of companies in the three simulation games relevant to this research-86 percent do competitive pricing analyses, 90 percent do competitive product studies, and 42 percent do competitive advertising studies. PRESENT EMPIRICAL STUDY Above, it is documented that the proposed PIMS criterion, as interpreted and applied in the G&F and Neal studies, is itself not a parameter of market systems. A necessary condition for adopting the criterion for the validation of simulation games is not satisfied. Additionally, the empirical research designs implemented by G&F and Neal are limited in several respects as will be explained below. In some instances, the limitations are demonstrated using data from The Marketing Management Experience (Dickinson 2002) which, as with Compete (Faria, Nulsen, and Roussos 1994) and Markstrat2 (Larréché and Gatignon 1990), is a competitive marketing management game. As with G&F’ study and the first of Neal’ replications, no budget constraint was in effect. Participants were MBA students enrolled in an introductory marketing course with each student individually managing a company. The 48 companies were grouped into eight industries of five companies each plus two industries of four companies each. (All industries in the G&F and Neal studies comprised five companies.) A competition of nine periods was administered. Company performance was measured using cumulative earnings, with students’ scores being based on that measure. Following the manipulation of the two previous studies, all possible reassignments of industry-leading companies were made, yielding 90 observations, compared with 25 for G&F and 12 for Neal. With each reassignment, the nine periods of competition were rerun using all companies’ original decisions. LIMITATIONS OF PREVIOUS EMPIRICAL RESEARCH DESIGNS The sole measure of strategy effectiveness in the G&F study is the ordinal position of cumulative earnings within each industry. Companies that were candidates for reassignment earned more than the other companies in their original industries and the performance of companies in the industries to which they were reassigned was also measured as first, second, and so on. Ordinal position is a fallible measure of effectiveness in that earnings for a reassigned company might well decline, i.e., its strategies might be less effective in the different competitive environment, yet not decline sufficiently to move it into second or lower place in the industry. Competitor strategies would have impaired the effectiveness of the focal strategy, yet the ordinal measure of effectiveness would not change. Neal measured actual profit in addition to ordinal position. Of the 12 cases in his study, eight reassignments resulted in lower profit than in the original 101 Developments in Business Simulation and Experiential Learning, Volume 30, 2003 industry. In the present study, described above, when industry leading companies were replaced with other industry leading companies, 26 of 90 (28.9%) again finished in first place and another 24 (26.7%) finished in second place. These percentages are lower than the corresponding results in the two earlier studies. However, the fallibility of ordinal position may still be starkly demonstrated. In fully 89 of the 90 reassignments (98.9%) did cumulative earnings drop in the industry to which the company was reassigned. Perhaps 55.6% of reassigned strategies placing first or second in a different competitive environment might be taken as moderate evidence of the independence of competitors’ strategies compared with, say, an approximate 40% chance probability of placing first or second. Where 98.9% of earnings are lower, though, the evidence of dependence is conclusive and the fallibility of ordinal position made clear. A second limitation of G&F’ research design is that the results are possibly idiosyncratic. Those researchers switched industry-leading companies among 25 industries. However, only one such reassignment was effected. Neal (and the present study), in contrast, reassigned all possible combinations of leading companies across four industries yielding 12 sets of results (90 in the present study). G&F’ results, then, may be to some extent or completely idiosyncratic. That is, their results may be peculiar to the single particular reassignment effected. Since Neal’ and the present study design switched all possible pairs, these results are not subject to this idiosyncracy. Possibly, in G&F’ design the percentage of instances in which a given strategy consistently placed first would be lower (or higher) had additional reassignments been carried out. The single reassignment design of G&F imposes an additional limitation; a fatal one. That is that virtually every possible outcome of their manipulation is consistent with their hypothesis. It is virtually impossible for their design to disprove their hypothesis. The basic conclusion of their research is that, since a high proportion of industry-leading companies in their original industries were also industry- leading in the industries to which they were reassigned, then the effectiveness of the strategies was largely unaffected by competitors’ strategies. A high proportion of “consistently good” strategies in that sense is taken by the researchers as indicating independence from competitor strategies. Would a low proportion of “consistently good” strategies, then, refute their hypothesis? No. Table 2 presents hypothetical cumulative earnings figures for five companies in each of four industries. In the scenario in Table 2, the leading company in Industry A replaces the leading company in Industry B. The leading company in Industry B replaces the leading company in Industry C and so on through the leading company in Industry D replacing the leading company in Industry A. An earnings figure is struck through in the original industry and is in italics in the industry to which it has been reassigned. Since independence from the competitive environment is assumed, none of the earnings figures changes as a result of the manipulation. TABLE 2: Hypothetical Reassignment of Industry-Leading Companies INDUSTRY A B C D 34 25 23 21 19 17 28 26 25 24 22 20 31 29 28 27 25 23 34 32 31 30 28 26 A strike-through indicates the earnings of a company in its original industry. Italics indicate the company’ earnings in the industry to which it was reassigned. As may be seen in Table 2, only a single company has the highest earnings in the industry to which it was reassigned. This despite competitor influence being absolutely nil. The scenario in the table is readily expanded to the 25 industries in the G&F study with the result that only 1 of the 25 original industry-leading companies, or 4 percent, would remain industry-leading in its new industry. With the manipulation conducted by G&F, any number of “consistently good” strategies from 1 through 25 could result under their hypothesis. The only possible result that could refute their hypothesis would be zero companies retaining their industry leading positions. Virtually any possible result of their study, including the one actually realized, would be inconclusive with respect to their hypothesis. This phenomenon appeared in Neal’ results. One company earned more profit in the industry to which it was reassigned than in its original industry yet ended in second position in the different competitive environment. By ordinal measure the effectiveness of the strategy decreased; by actual profit as a measure the effectiveness of the strategy increased. Further, “Rerunning the strategies with a different set of competitors consistently produced lower profit levels than in the original run of the simulation...The average level of profit achieved by the top two teams was substantially less than achieved in the original run”(Neal, p. 129). By ordinal measure, strategy effectiveness was consistent (all reassigned companies placed either first or second), while measured by profit, strategy effectiveness decreased. In Table 2, aggregate earnings levels in each industry are not equal. This conforms to G&F’ research. Their explanation (p. 34) as to why one of the reassigned companies fell from first to fourth or fifth place is that the overall industry demand was influenced by all companies in an industry and that the “inconsistent” company had found itself in an industry where its forecasts and shipping schedules were not appropriate. Even where overall industry demand does not change appreciably across industries, overall 102 Developments in Business Simulation and Experiential Learning, Volume 30, 2003 industry profit might still change based on the efficiency of companies’ strategies. Thus, the scenario in Table 2 is plausible in the context of G&F’ research design. The switching-leaders manipulation by G&F was replicated by Neal. A high proportion (72% or 18) of the 25 companies switched by G&F finished first in the industries to which they were reassigned, while an additional 12 percent or three companies finished second. Since in their new industries competitor strategies could not possibly have influenced the formulation of the reassigned strategies, the apparent implication is that the success of the switched strategies was independent of competitor strategies. Among the 12 switches effected by Neal, six finished first and six finished second in the industries to which strategies were reassigned, leading Neal to observe that, “This finding is consistent with the research of Green and Faria (1995)...indeed, it is even more compelling than their finding”(p. 123). The conclusion that in the respective games that competitor strategies have minimal impact on the effectiveness of the winning strategies does not follow from the switching- leaders manipulation. Placing an a priori effective strategy into a mix of a priori ineffective competitor strategies does not test whether competitor strategies are incorporated into the games’ algorithms. (It is this incorporation or, rather, the absence of this incorporation, that is the basis for G&F’ PIMS- based criterion for validity.) In the switching-leaders design, strategies are dichotomized as effective (i.e., industry leading) or ineffective (i.e., not industry leading) and the manipulation merely switches strategies of the same type. An a priori effective strategy is grouped with a priori ineffective strategies under all reassignments. The fallacy of the switching-leaders design is readily demonstrated. A more appropriate design would place an a priori effective strategy into a relatively effective mix of competitor strategies rather than into a relatively ineffective mix. If a focal strategy is effective independent of competitor strategies, then whether competitor strategies are relatively effective or ineffective should have no bearing on the focal strategy. In the present study the switching-leaders manipulation was applied as already described. A second type of manipulation, though, demonstrates the invalidity of those results and the results from the previous studies as indicating the extent of competitor-(in)dependence of strategy effectiveness. The second manipulation did not replace the leading strategy in one industry with the leading strategy from another industry. Rather, the leading strategy in one industry replaced the most ineffective, i.e., least cumulative earnings, strategy in a second industry. A leading strategy was placed into a mix of relatively effective competitor strategies, that is, relative to the mix where the leading strategy was displaced from the competitor mix as with the earlier studies. If the effectiveness of the leading strategy is independent of competitors, then the a priori effective strategy that was reassigned should still be effective. Of the 90 such reassignments, 56 (62.2%) of the strategies finished last. An additional 19 (21.1%) of the strategies finished next to last. In only a single instance (1.1% versus 28.9% under the switching-leaders manipulation) was the most effective strategy in its original industry also the most effective in the industry to which it was reassigned and in only seven instances (7.8% versus 26.7%) was it the second most effective. Under the switching-leaders manipulation, 55.6% of the reassigned companies finished first or second in the industries to which they were reassigned. Under the switching-leaders- and-losers manipulation, the corresponding result is 8.9%. The incontrovertible difference in the proportions of “consistent” strategies demonstrates the internal invalidity of the former manipulation. The research designs of G&F and of Neal did not incorporate this more rigorous manipulation and, as such, their results cannot speak to the conformance of the games studied to a competitor-independent validity criterion (which criterion, as evidenced earlier, is invalid in itself). STRATEGIES VERSUS DECISION VALUES In analyzing the results of his extended study, Neal raised “...the fundamental question as to whether [his] research and that of Green and Faria (1995) are actually testing strategies or simply replicating decisions out of context”(p. 129). In other words, in formulating their original strategies, did managers of successful companies in fact largely or entirely ignore competitor strategies, while managers of less successful companies took greater account of competitor strategies? The data for the present study provide a direct behavioral indicator of the comparative extent to which the two groups of managers-successful and unsuccessful- considered competitor strategies in formulating their own strategies. The Marketing Management Experience (Dickinson 2002) makes available for purchase eleven marketing research reports. Eight of these reports present the strategies of competitors for respective marketing mix elements, e.g., prices, advertising expenditures and messages, etc. If managers of industry-leading companies formulated their strategies less on the basis of competitors’ strategies than did managers of less successful companies, then there would be no benefit in their purchasing marketing research that reports on competitors’ strategies. Managers of industry-leading companies purchased an average of 24.6 research reports of competitors’ strategies over the course of the nine-period competition. Managers of companies that finished last in their industries purchased an average of 22.7 reports. For the last five periods of the competition (few reports are purchased in the final period as there are no further strategies to be formulated) managers of industry-leading companies purchased an average of 10.9 research reports of competitors’ strategies. Managers of companies that finished last in their industries purchased an average of 9.5 reports. That managers of industry-leading companies purchased substantial numbers of reports of their competitors’ strategies strongly indicates that they considered that information in formulating their strategies. That they purchased marginally 103 Developments in Business Simulation and Experiential Learning, Volume 30, 2003 more and, obviously, not substantially fewer such reports than industry-trailing companies makes that indication more conclusively. CONCLUSION Validation of simulation games is a necessary and worthwhile pursuit. The extensive and carefully developed PIMS data base is a potentially appealing source of parameters for validation. However, Green and Faria (1995) and Neal (1999) adopted an ostensible PIMS finding uncritically and erroneously. The operationlization of the PIMS data base itself, principles derived from analysis of those data, results of numerous studies making use of the PIMS data, and abundant complementary theory and basic research render competitor- independence of marketing strategy effectiveness unworthy as a validation criterion. Researchers considering using a simulation game should decidedly not select one where the effectiveness of marketing strategy is independent of competitors. Incidentally, the present study demonstrates that strategy effectiveness in The Marketing Management Experience (Dickinson 2002) is, validly, partially a function of competitors’ strategies. Green and Faria (1995) and Neal (1999) seem to have demonstrated that the Compete (Faria, Nulsen, and Roussos 1994) and Markstrat2 (Larréché and Gatignon1990) games meet an invalid criterion. However, their research design is invalid for that purpose and there is nothing in their studies or the present one that impugns the validity of the latter two games. REFERENCES Aaker, David A. (1998), Strategic Marketing Management, Fifth Edition (New York: John Wiley & Sons, Inc.). Abell, Derek F. and Hammond, John S. (1979), Strategic Market Planning (Englewood Cliffs, NJ: Prentice-Hall, Inc.). Buzzell, Robert D. and Gale, Bradley T. (1987), The PIMS Principles (New York: The Free Press). Buzzell, Robert D. and Wiersema, Frederik D. (1981), “Successful Share-Building Strategies,” Harvard Business Review, Vol. 59 (January-February), 135-144. Cohen, Kalman J. and Rhenman, E. (1961), “The Role of Management Games in Education and Research,” Management Science, Vol. 7 (January), 131-166. Dickinson, John R. (2002), The Marketing Management Experience (Windsor, Ontario: Management Experiences). Faria, A. J., Nulsen, R. O., and Roussos, D. S. (1994), Compete (Burr Ridge, IL: Richard D. Irwin, Inc.). Green, Donna and Faria, A. J. (1995), “Are Good Strategies Consistently Good?” in Overby, John D. and Patz, Alan L. (editors), Developments in Business Simulation & Experiential Exercises, Volume 22 (Statesboro, GA: Association for Business Simulation and Experiential Learning), 31-37. Grewal, Dhruv, Kavanoor, Sukumar, Fern, Edward F., Costley, Carolyn and Barnes, James (1997), “Comparative Versus Noncomparative Advertising: A Meta-Analysis,” Journal of Marketing, Vol. 61 (October), 1-15. Hanssens, Dominique M., Parsons, Leonard J., and Schultz, Randall (2001), Market Response Models, Second Edition (Boston: Kluwer Academic Publishers). Hawkins, Del I., Best, Roger J. and Coney, Kenneth A. (2001), Consumer Behavior, Eighth Edition (New York: Irwin/McGraw-Hill). House, William C., Parks, Don M., and Lindstrom, Grant L. (1994), “Relationships between R&D and Profitability: An Exploratory Comparison of Two Business Simulations with Two Real-World, Technology Intensive Industries,” in Thavikulwat, Precha and Overby, John D. (editors), Developments in Business Simulation & Experiential Learning, Volume 21 (Statesboro, GA: Association for Business Simulation and Experiential Learning), 75-78. House, William C. and Taylor, Lewis A. (1992), “The Lagged Effects of Decision Variables on Financial Performance Measures Used in Two Business Simulations,” in Gosenpud, Jerry and Gold, Steven (editors), Developments in Business Simulation & Experiential Exercises, Volume 19 (Statesboro, GA: Association for Business Simulation and Experiential Learning), 213. Jain, Subhash C. (2000), Marketing Planning & Strategy, Sixth Edition (Cincinnati, OH: South-Western College Publishing). Kinnear, Thomas C. and Root, Ann R. (editors) (1995), 1994 Survey of Marketing Research (Chicago: American Marketing Association). Kotler, Philip, Cunningham, Margaret H. and Turner, Ronald E. (2001), Marketing Management, Canadian Tenth Edition (Toronto: Prentice Hall). Larréché, Jean-Claude and Gatignon, Hubert (1990), Markstrat2: A Marketing Simulation Game (Palo Alto, CA: Scientific Press). Meurs, Freddy & Choffray, Jean-Marie (1975). Business games: Their role in training and development. Bradford, England: MCB Monographs. Napier, Herman S. and House, William C. (1990), “Simulated Versus Actual Firms: A Ratio Interaction Matrix Comparison,” Simulation & Gaming: An International Journal, Vol. 21 (June), 166-180. Naylor, Thomas H., Balintfy, Joseph L., Burdick, Donald S., and Chu, Kong (1966), Computer Simulation Techniques (New York: John Wiley & Sons, Inc.). Neal, Derrick J. (1999), “How Consistent are Winning Strategies? The Role of Competitor Analysis and Budgets on Performance in a Simulation,” Simulation & Gaming: An International Journal, Vol. 30 (June), 118-131. Parsons, Leonard J. and Schultz, Randall L. (1976), Marketing Models and Econometric Research (New York: North- Holland Publishing Company). Schoeffler, Sidney (1993), Basic Findings on Business Strategy (Cambridge, MA: The Strategic Planning Institute). 104 Developments in Business Simulation and Experiential Learning, Volume 30, 2003 Schoeffler, Sidney (1983), “The PIMS Program” in Albert, Kenneth J. (editor), The Strategic Management Handbook (New York: McGraw-Hill Book Company), 23-1-23-10. Schoeffler, Sidney (1977), A Cross-Sectional Study of Strategy, Structure, and Performance: Aspects of the PIMS Program” in Thorelli, Hans B. (editor), Strategy + Structure = Performance (Bloomington, IN: Indiana University Press), 108-121. 105 Developments in Business Simulation and Experiential Learning, Volume 30, 2003 106 Thavikulwat, Precha (1988), “Simulating Demand in an Independent-Across-Firms Management Game,” in Sanders, Patricia and Pray, Tom (Eds.), Developments in Business Simulation and Experiential Learning, Volume 15 (Statesboro, GA: Association for Business Simulation and Experiential Learning), 183-187. Table of Contents Volume 30, 2003 The Optimal Timing For Introducing Business Simulations Can Handicapped Students Access Your Class Web Site? The Competition Game: Decision Making In A Dynamic Environment Pan-Pacific Enterprises: Strategic Decision Making Simulation Study Of Stochastic Channel Redistribution The Impact Of Business War Games: Quantifying Training Effectiveness Experiential Learning: Introducing Faculty And Staff To A University Leadership Development Program The Feasibility Of The Balanced Scorecard For Business Games A Misuse Of Pims For The Validation Of Marketing Management Simulation Games Incorporating Technology Into The 21st Century Classroom: Are We Facilitating Academic Dishonesty? Improving The Effectiveness Of Peer Evaluations The Use Of A Simulation In An Integrated Mba Curriculum Student Portfolios In Business Education Student Portfolios In Business Education At Ashland University Using SAP ERP Technology To Integrate The Undergraduate Business Curriculum Board Games And Teaching Textile Marketing And Finance Blogging: A New Threat To Student Research? The Way We Talk! Take II Strategic Management: An Evaluation Of The Use Of Three Learning Methods In Hong Kong Adoption Of Discussion-Based Teaching And Assessment In Teaching Strategic Management In Hong Kong The Tobin Q As A Company Performance Indicator Using Representative Nominal Group Technique For Course Review And An Interactive Solicitation Of Ways To Enhance Absel's Image Making Teaching Matter: The Art And Science Of Teaching Business Communication Beyond Sex, Age, And Race: Exploring The Deeper Contents Of Diversity Teaching & Learning The Facilitation Process A Brief On Debriefing: What It Is And What It Isn't Changing Perceptions Of The Importance Of Leadership: The Contribution Of Individual Spirit Harmonics In Leadership Knowledge, Skills And Sustainable Values In Learning Organizations: Some Implications From The Multicultural Virtual Classroom Challenges Of Teaching Undergraduate Organizational Behavior In A Nontraditional Time Format Interactive Online Positioning With The Web-Based Product Positioning Map Graphics Package The Longitudinal Effects Of Entrepreneurship Training On Risk Tolerance: A Look At Similarities And Differences Between Male And Female Undergraduate Students Revisiting Strategy Learning In A Total Enterprise Simulation What Are Simulations For?: Learning Objectives As A Simulation Selection Device Gaming Agency Markets Cooperate For Profits Or Compete For Market? Study Of Oligopolistic Pricing With A Business Game The Design Of A Business Simulation Using A System-Dynamics-Based Approach Modeling The Product Development Function For An Entrepreneurial Firm Simulation Performance And Forecast Accuracy? Is That All? Business Manager Identification Of Competitors In Real World And Simulation Settings Monte Carlo Simulation Analysis On The Costs Reduction Argument Of Interest Rate Swaps Ebiz Game: A Scalable Online Business Simulation Game For Entrepreneurship Training A Model For Online Education Delivery And A Look At Online Delivery Effectiveness Incorporating "Company Reputation" into Total Enterprise Simulations The Genesis And Future Of The Absel "Classicos" Initiative