The Invalidity of the "Natural Market Structure" PIMS Validation of Simulation Game 124 Developments in Business Simulations and Experiential Learning, Volume 37, 2010 THE INVALIDITY OF “NATURAL MARKET STRUCTURE” PIMS VALIDATION OF SIMULATION GAME John R. Dickinson Management Experiences MExperiences@bell.net ABSTRACT The Profit Impact of Market Strategies (PIMS) database, prominent in the 1970s and 1980s, has provided the basis for a series of works that would validate business games vis-a-vis PIMS. Owing to some limitations of PIMS, business games and PIMS not being comparable in some critical respects, and misinterpretations of PIMS, the present paper examines a recent one of the series of PIMS- based validation studies. INTRODUCTION “Validating Business Simulations: Do Simulations Exhibit Natural Market Structures?” (Wellington & Faria 2006, hereafter W&F) puts forth two criteria for the validation of simulation games. The study draws on the PIMS project for real-world “principles” against which business games might be validated. Citing but a single reference, Buzzell 1981, the validation criteria put forth are a certain structure of company market shares and a certain market share size ratio, the two criteria complementing each other. This paper establishes the invalidity of those two criteria. PIMS NON-REPRESENTATIVE SAMPLE OF BUSINESSES Robert Buzzell was one of the first research directors of the PIMS project. In the same sole reference cited in the paper, Buzzell declares the non-representativeness of PIMS businesses: “These companies [those participating in the PIMS research program] obviously do not constitute a representative sample of all business firms.” (Buzzell 1981, p. 45). It would be difficult, if not impossible, for any sample to meaningfully represent all business firms. Granting that, still begged is the issue of how a non- representative collection of businesses can constitute a source of reality game-validation criteria, an issue fatal to all PIMS-based “validation” criteria. INFEASIBILITY OF REPLICATING PIMS MARKET SHARE IN A GAME As noted in the “Introduction,” both of the two “validation” criteria put forth are based on simulation company market shares. Fatally, it is not feasible to replicate PIMS market share in a simulation game for at least two fundamental reasons. PIMS ELICITATION OF MARKET SHARE “The PIMS data differ from industry data commonly used in studies of business firm size and market structure in one important respect. Each business unit reports its own sales and market share annually, and estimates of the market shares of its principal competitors. These market shares are measured in relation to the business unit’s served market...Given that each business unit develops its own definition of its served market, it is inevitable that the PIMS market share data are not based on a uniform concept of what constitutes a ‘market’.” ( Buzzell 1981, p. 45) Market share definitions and own and competitor market share size estimates in PIMS are idiosyncratic to the responding company. In contrast, in simulation games definitions of business units and markets are clear, fixed, and uniform. At least within the state of contemporary business games, it is infeasible that individual participants could define business units and served markets peculiar to their own companies. Buzzell (1981, p. 45) rationalizes that there may be no way to avoid ambiguous market boundaries and “...that the PIMS data are at least as good if not better than those available from other sources...” With respect to validation of simulation games, though, that PIMS data might satisfice for some purposes is not the point. The point is that it is infeasible to replicate the PIMS approach. Specifically, in the example applications (W&F, Tables 1 and 2, pp. 121- 122), the PIMS approach is not faithfully replicated. With this, any findings from PIMS that incorporate market share cannot serve to validate a simulation game. This invalidity holds for the other purported PIMS-validation papers in the series. INVENTORY STOCKOUTS Dickinson (2006) has argued that sales, and therefore market shares, in simulation games may not be comparable to sales in PIMS. In simulation games where the decision mix includes inventory management, sales may not equal 125 Developments in Business Simulations and Experiential Learning, Volume 37, 2010 demand due to stockouts. Inventory is managed in many real-world companies, too, of course. However, there is no reason to believe that the frequency of stockouts and the extent of unsatisfied demand are comparable between a given simulation game and the multitude of firms involved with PIMS. Dickinson cited game competitions comprising approximately 6,000 inventories where between 20.9 and 27.3 percent stocked out, a level surely rare in the established businesses (“...most of the firms represented are members of the 1979 Fortune 500 list.”, Buzzell 1981, p. 45) that comprise PIMS. In games not requiring inventory management, stockouts may not be a factor, an opposite extreme just as unlikely in PIMS companies. “In short, stockouts almost certainly affect game sales (and profits) differently than PIMS sales (and profits).” (Dickinson 2006) PIMS-INVALID MODEL The authors of “Validating Business Simulations: Do Simulations Exhibit Natural Market Structure?” make clear that it is the SPSS logarithmic model that is the PIMS model against which simulation games should be validated. Their first hypothesis, H1, is that, “The size distribution of companies competing in different industries of the MERLIN Marketing Simulation will fit a logarithmic distribution...” (p. 120) The paper’s Table 1 (p. 121, reproduced below) contains 10 of the 11 models (the inverse model being excluded) estimated by the SPSS Curvefit procedure and the first line of results in that Table 1 is labeled “Logarithmic (PIMS)” the parenthesized “PIMS” being theirs. In contrast (and contradiction), Buzzell’s natural market structure model is semi-logarithmic. Specifically, Buzzell’s express model is Log MSi = k0 + k1 (Rank) where MSi is the market share of the i-th largest competitor (1981, p. 43). As Buzzell makes clear the model he estimated, SPSS makes just as clear its model definitions. The SPSS Curvefit logarithmic model is Y = b0 + b1 ln(t) (SPSS, p. 238) which is not the model Buzzell estimated. The SPSS Curvefit growth model, ln(Y) = b0 + b1(t) is the model Buzzell estimated. By the results reported in Table 1 (p. 121, reproduced below), the growth model is the fifth best fitting model (tied with three other models). This is the case for each of the 12, 9, and 6 team results reported in the paper. In other words, the actual Buzzell model is no better a fit to the example game data than seven different models. -------- insert table 1 about here -------- Other than its being listed in Table 1, no mention of the SPSS growth model, i.e., Buzzell’s actual model, is made. In sum, the suggested validation criterion is simply not Buzzell’s model of “natural market structure.” There is no basis whatsoever in PIMS for the logarithmic game- validation criterion. PIMS-IMPOSSIBLE APPLICATIONS The authors illustrate their PIMS-invalid natural market structure criterion using market shares of competitions comprising 19 industries of 12 companies each, 20 industries of 9 companies each, and 20 industries of 6 companies each (p. 120). That those market shares do not replicate those operationalized in PIMS has been explained above. Further, though, it is impossible to derive any market share related findings from PIMS for shares of any more than four companies. PIMS elicitation of market shares is limited to that of the responding company and those of the company’s three largest competitors (Abell & Hammond 1979, p. 311). There are no market share data, however operationalized, in PIMS for any more than four competing companies. With respect to Buzzell’s (1981) natural market structure analysis, it is also impossible to derive findings for any fewer than four companies: “For purposes of analyzing distributions of market shares, the data base has been Table 1: Wellington and Faria Model Estimations 12 Teams (N-217) 9 Teams (N=173) 6 Teams (N=119) Curve Distribution R square Sig R square Sig R square Sig Logarithmic (PIMS) Linear Quadratic Cubic Compound Power S-Curve Growth Exponential Logistic .747 .731 .745 .759 .713 .660 .497 .713 .713 .713 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .646 .682 .664 .689 .618 .593 .488 .618 .618 .618 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 .699 .667 .689 .708 .602 .579 .499 .602 .602 .602 .000 .000 .000 .000 .000 .000 .000 .000 .000 .000 Source: Wellington and Faria (2006, p. 121). Italics indicate Buzzell’s (1981) model. 126 Developments in Business Simulations and Experiential Learning, Volume 37, 2010 restricted to those markets in which there were at least four significant competitors...at least four data points are required to fit a distribution to each market” (p. 46). Between the maximum four companies in the original PIMS data elicitation and the minimum of four data points required for model estimation, any implications for simulation games of Buzzell’s natural market structure analysis apply only to market shares of exactly four companies. Buzzell further limited his selection of PIMS companies to those where the market shares were those of the four leaders (1981, p. 46). EQUIVOCAL MODEL; EQUIVOCAL VALIDATION CRITERION As noted above, the paper’s first hypothesis specifically stipulates a logarithmic model and, as also explained above, that is not the model for which Buzzell reports results. Further, Buzzell is candid that even the results he does report are equivocal. “In evaluating these results it must be recognized that with only four data points, any reasonable type of distribution would provide a ‘good’ fit...Thus, it cannot be argued that this particular distribution [semi- logarithmic] is uniquely appropriate as a description of the distribution of relative size among competitors in individual product markets.” (Buzzell 1981, p. 46) The accommodating implication of Buzzell’s equivocation is specious: any reasonable type of distribution would validate a simulation game. PIMS-INVALID DATA AGGREGATION As mentioned above, the paper reports results for 10 of the 11 SPSS Curvefit models. For each model the sample size is 217 (~19 industries of 12 companies each) or 173 (~20 industries of 9 companies each) or 119 (~20 industries of 6 companies each). That is, data for all 12-company industries were aggregated, as were data for all 9-company industries and for all 6-company industries, respectively. (It has previously been documented that PIMS has no data for market shares for more than four companies, not 12 or 9 or 6 companies.) Those, then–217, 173, 119–are the number of data points on which each model and each R2 was estimated in the paper. This is not the level of data aggregation used by Buzzell. Each of the R2s reported by Buzzell was for but the four data values elicited from a single company for a single product-market, a total of 1,218 separate R2s. The authors do not mention the average of Buzzell’s R2s (0.851, p. 46) which is considerably greater than the R2s for the PIMS-relevant SPSS growth model (0.713, 0.618, 0.602) for the example game data. For that matter, Buzzell’s average R2 is considerably greater than those for the logarithmic model that the authors incorrectly attribute to PIMS (0.747, 0.646, 0.699). Nor do the authors mention the distribution of those 1,218 R2s reported by Buzzell (1981, Table 1, p. 46). In light of that distribution, the mean (0.851) is disqualified as a standard for validation. PIMS yields R2s over the entire range of possible values, not a single parameter. For 4.3 percent of the 1,218 PIMS companies the semi- logarithmic model is a relatively poor fit with R2s of 0.5 or less (Buzzell 1981, Table 1, p. 46). Are those low R2s any less valid than higher R2s? Are those company market shares somehow less valid than other PIMS company market shares? To assert so would be to invalidate data on the basis that they do not conform to an hypothesis rather than vice versa! PIMS-INVALID SIZE RATIO The so-called size ratio concept posits that a company’s market share is a constant proportion of the share of the next higher-ranking company’s market share. Regarding the size ratios in their data (Table 2, p. 122), the authors claim, “The finding that size ratios had a consistent value is in line with PIMS data although Buzzell (1981) indicates that the size ratio from the PIMS database averaged 0.6 while ranging ‘from a low of 0.42 for the automobile industry to a high of 0.89 for beer and gasoline.’” (p. 121). Also, “...the size ratios in the simulation were at the higher range as compared to real world industries (in the range of .90).” (p. 118) The authors’ claim is incorrect in at least three vital respects. First, the paper’s example data are from the Merlin (Anderson et al. 2004) simulation and as the authors note (p. 120), “The simulation itself designates the products generically as Product 1 and Product 2...” As such, the Merlin products have no specific real-world counterparts. Are Product 1 and Product 2 more like automobiles (size ratio 0.42) or more like beer and gasoline (size ratio 0.89)? It is impossible to know what portion of the 0.42-0.89 range might be relevant to the Merlin game. Generally, the entirety of the PIMS project is completely irrelevant to any simulation game using generic products. Second, the common basis of the several invalid PIMS- validation works that have been published is to draw from PIMS real-world criteria against which simulation games might be validated. But the 0.42-0.89 size ratio range referred to by the authors was not derived from the PIMS project. Buzzell (1981, p. 43) clearly attributes the 0.42- 0.89 range to Cooke and Cox (Cox 1977), a study entirely separate from PIMS. The authors erroneously attribute the 0.42-0.89 range to PIMS and make no mention of the actual PIMS range of size ratios (Buzzell 1981, Table 1, p. 46). Third, by its definition a size ratio must be between zero and one. And, indeed, the distribution of size ratios for 1,218 PIMS business units runs that gamut, 9.1% being below 0.4 and 1.6% being above 0.9 (Buzzell 1981, Table 1, p. 46). It is impossible for size ratios from any simulation game to not lie between zero and one. The authors’ PIMS- based size ratio for game validation is moot, i.e., invalid. 127 Developments in Business Simulations and Experiential Learning, Volume 37, 2010 ARBITRARY GAME (IN)VALIDATION The authors employ two statistics–R2 (Table 1, p. 121) and size ratio (Table 2, p. 122)–related to their two hypotheses (p. 120), respectively. Each of the two statistics, coincidentally, by definition has a range of zero to one. As noted above, in his Table 1 Buzzell (1981, p. 46) presents PIMS-based R2 values throughout the possible range of zero to one and also in that table presents PIMS-based size ratio values throughout the possible range of zero to one. Consider an R2 value and a size ratio value calculated for a given simulation game. In light of the non- representativeness of PIMS businesses and the infeasibility of replicating PIMS’ market share operationalization in a business game explained above, per se neither game-based statistic would be a valid criterion. Beyond that, though, a resulting value for each statistic for the game must by definition lie between zero and one. That is, it must lie within some portion of the corresponding PIMS distribution in Buzzell’s Table 1 and it must lie outside other portions of the PIMS distribution. The game could be deemed valid by referring to the portion of the PIMS distribution that includes the value and the game could be deemed invalid by referring to portions of the PIMS distribution not including the value. The game could be deemed valid or invalid arbitrarily. CONCLUSION The authors put forth two ostensible PIMS-based criteria for simulation game validation. The first criterion is a logarithmic model for market share rank when, in fact, the only source the authors cited in support of that criterion, i.e., Buzzell 1981, uses a different model (i.e., a semi-log or growth model). The second criterion is a market share size ratio when PIMS size ratios are distributed across the entire possible range of values and, thus, the ratio calculated for any simulation game data must necessarily lie within that range. The potential consequences of the invalidity of the authors’ hypotheses and the entire series of PIMS-invalid published criteria are not benign. Had the authors correctly interpreted their empirical results (those results themselves being invalid), the Merlin game would have been deemed invalid. No fewer than seven of the reported structural models fit the data as well as or better than Buzzell’s model. The reported size ratios of about 0.9 are PIMS-atypical (only 1.6% of the PIMS ratios are above 0.9) and are well off the PIMS average of 0.64 (that average itself being a meaningless standard, given there is a wide distribution of PIMS-valid values). Games not meeting the raft of published invalid PIMS “validation” criteria are unfairly compromised as are administrators and participants who would erroneously eschew those games. Ahead, game developers might design games toward the invalid PIMS “validation” criteria, unwittingly producing invalid games. Additional basic research might be–and has been (Edman 2006, Pillutla & Thavikulwat 2005, Thavikulwat 2005)–founded on the extant invalid published works, misdirecting other researchers’ efforts and further impairing the store of knowledge. Evaluating simulation gaming in many facets, of which game validity is one, is a prominent concern of the field; Feinstein and Cannon (2002) provide a comprehensive overview. It is, of course, vital that criteria for the evaluation of game validity themselves be valid. REFERENCES Abell, Derek F. and John S. Hammond (1979). Strategic Market Planning, (Englewood Cliffs, NJ: Prentice-Hall, Inc.). Anderson, P. H., D. A. Beveridge, L. Lawton, and T. Scott (2004). Merlin: A Marketing Simulation (New York: McGraw-Hill/Irwin). Buzzell, Robert D. (1981). “Are There ‘Natural’ Market Structures?” Journal of Marketing, Vol. 45 (Winter), 42-51. Cox, William E., Jr. (1977). “Product Portfolio Strategy, Market Structure, and Performance,” in Thorelli, Hans. B. (editor), Strategy + Structure = Performance (Bloomington, Indiana: Indiana University Press), 83- 102. Dickinson, John R. (2006). “The Invalidity of Profit=f(Market Share) PIMS ‘Validation’ of Marketing Games,” in Smith, J. Alexander (editor), Developments in Business Simulation and Experiential Learning, Volume 33 (Association for Business Simulation and Experiential Learning), 232-237. Reprinted in the Bernie Keys Library [Available from http://www.ABSEL.org] Edman, Jan (2006). “Group Composition and Groupthink in a Business Game,” in Smith, J. Alexander (editor), Developments in Business Simulation and Experiential Learning, Volume 33 (Association for Business Simulation and Experiential Learning), 278-283. Reprinted in the Bernie Keys Library [Available from http://www.ABSEL.org] Feinstein, Andrew Hale and Cannon, Hugh M. (2002), AConstructs of Simulation Evaluation,@ Simulation & Gaming, Vol. 33 (December), 425-440. Pillutla, Sharma and Precha Thavikulwat (2005). “Forecasting Stock Value,” in Ledman, Robert E. (editor), Developments in Business Simulation and Experiential Learning, Volume 32 (Association for Business Simulation and Experiential Learning), 255- 261. Reprinted in the Bernie Keys Library [Available from http://ABSEL.org] SPSS (2001). SPSS 11.0 Syntax Reference Guide, Volume 1 (Chicago: SPSS Inc.). Thavikulwat, Precha (2005). “Effect of Market Share and Production Experience on Company Profitability,” in Ledman, Robert E. (editor), Developments in Business Simulation and Experiential Learning, Volume 32 128 Developments in Business Simulations and Experiential Learning, Volume 37, 2010 (Association for Business Simulation and Experiential Learning), 299-306. Reprinted in the Bernie Keys Library [Available from http://ABSEL.org] PIMS-INVALID “VALIDATION” PUBLICATIONS Faria, A. J. and William J. Wellington (2005). “Validating Business Gaming: Business Game Conformity with PIMS Findings,” Simulation & Gaming, Vol. 36 (June), 259-273. Faria, A. J. and William J. Wellington (2005). “Validating Business Simulations: Does High Product Quality Lead to High Profitability?” in Ledman, Robert E. (editor), Developments in Business Simulation and Experiential Learning, Volume 32 (Association for Business Simulation and Experiential Learning), 117-121. Reprinted in the Bernie Keys Library [Available from http://ABSEL.org] Faria, A. J. and William J. Wellington (2004). “Validating Business Simulations: Does High Market Share Lead to High Profitability?” in Feinstein, Andrew Hale and Potosky, Denise (editors), Developments in Business Simulation and Experiential Learning, Volume 31 (Association for Business Simulation and Experiential Learning), 332-336. Reprinted in the Bernie Keys Library [Available from http://ABSEL.org] Green, Donna and A. J. Faria (1995). “Are Good Strategies Consistently Good?” in Overby, John D. and Patz, Alan L. (editors), Developments in Business Simulation & Experiential Exercises, Volume 22 (Association for Business Simulation and Experiential Learning), 31-37. Reprinted in the Bernie Keys Library [Available from http://ABSEL.org] Neal, Derrick J. (1999). “How Consistent are Winning Strategies? The Role of Competitor Analysis and Budgets on Performance in a Simulation Game,” Simulation & Gaming, Vol. 30 (June), 118-131. Wellington, William and A. J. Faria (2006). “Validating Business Simulations: Do Simulations Exhibit Natural Market Structures?” in Smith, J. Alexander (editor), Developments in Business Simulation and Experiential Learning, Volume 33 (Association for Business Simulation and Experiential Learning), 118-123. Reprinted in the Bernie Keys Library [Available from http://ABSEL.org] Table of Contents Volume 37, 2010 A Simulation in Organizational Behavior: Development and Beta Test Simulating Networks: An Experiment Pick Your Group Size: A Better Procedure to Resolve the Free-Rider Problem in a Business Simulation The Impact Of Playing A Marketing Simulation Game On Perceived Decision Making Ability Among Introductory Marketing Students If the Games Work, Why Aren't More Faculty Willing to Play? Modeling Cascading Demand: Accounting for the Effects of Captive Consumer Relationships Issues In Simulation Implementation: Lessons From a Freshman Seminar ENDURANCE: A game for current economic times Simulating Processes: An Application In Supply Chain Management Another Look At The Use Of Forecasting Accuracy On The Assessment Of Management Performance In Business Simulation Games The Effect Of Advertising On Demand In Business Simulations The Relationship Between Learning And Performance In A Total Enterprise Simulation: Revisited And New Data. Is the Gold /Pray Simulation Demand Model Valid and is It Really Robust? Mission Possible - Using Simulation Games For Management Training In A Transition Economy Business Game Modeling For The Negotiation Between General Contractor And Subcontractors The Relationship Between Goal Orientation And Simulation Performance With Attitude Change And Perceived Learning: A Follow-Up Study The Invalidity Of "Natural Market Structure" Pims Validation Of Marketing Games Examining A Beta Test An Experiment On Group Decision-Making Using A Business Game: An International Comparison Of Mba Students In Japan; China; And Russia A Platform for Business Games - From Game Playing to Game Making: The Case of Yokohama National University Checking Financial Balance of Target Brand Portfolio with the Strategic Market Plan Cash Flow Package About Simulations and Bloom's Learning Taxonomy Managing Complexity: Applying the Conscious-Competence Model to Experiential Learning Using Accumulated Profits to Assess Performance in Simulations And Now Let's Really Innovate: The Notion of Perpetual Online Courses Distance Learning: A Game Application Web Based Total Enterprise Simulation Learning Business Administration Using Simulation Software A Review of Experiential Exercises in Absel's Published Proceedings Moving Outside the Classroom: An Experiential Exercise Involving Video Clips Peer To Peer: A New Tool for Student Peer Evaluation Continuing To Assess Project Management: What's Been Done? What's It Means? Looking At New Twists Strategies for Promoting Knowledge Formation and Dissemination in Business Simulation and Experiential Learning Introducing Experiential Exercises To Inexperienced Faculty Members Conveying Concepts Through Cartoons: Preliminary Results From Software Engineers In An Industrial Organization Theory Class Utilizing Institutional Problems as an Experiential Learning Opportunity for Students Considerations in the Design of Vicarious Learning Environments Enhancing Critical Thinking Through Service-Learning As A Consultative Process Encouraging "Established" Faculty to Adopt a "New" Games and Simulations Pedagogy IMA Candy Company: An Experiential Exercise in Managerial Accounting Mustard Seeds as Means for Creative Problem Solving, Ethical Decision Making, Stakeholder Alliance, & Leader Development through Experiential Learning in Management Education Interactive Classroom Technology: Adding a Benefit to the Traditional Classroom Excelling in the Capstone Simulation: An Application of Spreadsheet Technology as a Decision Support Tool GEO: An Entrepreneurship-Oriented Computer-Assisted International Strategy Simulation ERP Simulation Game: A Distribution Game to Teach the Value of Integrated Systems