MUSINGS ON BUSINESS GAME PERFORMANCE EVALUATION Developments In Business Simulation & Experiential Learning, Volume 24, 1997 MUSINGS ON BUSINESS GAME PERFORMANCE EVALUATION Hans B. Thorelli, Indiana University ABSTRACT In special-purpose business games as well as in numerous simple general management games, a single indicator is often appropriate for evaluating the performance of participating teams. Typically, but far from always, the nature of the performance indicator is fairly self-evident, thought its ultimate rationale may be relatively weak. In more complex functional game and especially in intricate general management simulations, the need for multimeasurement grows, perhaps even faster than the complexity itself This becomes especially evident in games that permit truly entrepreneurial decisions, with concomitant variation in philosophy, scope, and scale of individual companies. INTRODUCTION Many business games designers and users believe that a single measure should be used to evaluate team performance. Analogous to gauging the temperature of water m a pool, it certainly simplifies score-keeping! Assuming that students have been alerted in advance, the single- (measurement)-minded would also claim that clinging to a single scale has an inherent element of fairness: whether participants agree with the measure or not, they “know what the score is." Just like in a horse race, there would generally be no doubt that about the winner. And, indeed, a majority of games are as simple in concept as horse races; thus, a “unimeasure” may be quite justified. In such cases, inherent logic will often make the specification of the measure fairly obvious. Regardless of how performance is being measured, an important question is always whether the assessment will reflect a better understanding of the “reality” being simulated, or whether it merely reflects superior ability in “beating the game.” Unless the issues in the game are narrowly defined, and fairly “technical” in nature, the risk is substantial that the score mirrors understanding of the game model rather than the richness of reality. Neoclassical economists might say that a good, single performance measure is the stock price of a company. This is a contentious assertion. Stock price does not have the merit of a single gauge, but it is dependant on a host of independent variables, such as dividend to earnings ratio, growth in sales, market share, profitability, riskiness, and future prospects. Clearly, some of these variables are difficult to define unambiguously-not to speak of the difficulty and/or complexity of measuring them. Further, the relative weight to assign the component variables is essentially an arbitrary matter. In effect, stock price is actually a multidimensional measure, and one whose rationale in a given game model may be difficult to specify, let alone to make credible. COMPLEX GAMES MAKE FOR COMPLEX EVALUATIONS A growing group of overall strategy and other general management simulations are characterized by increasing complexity. As every Wall Street analyst knows, complexity calls for consideration of a portfolio of indicators. Different analysts tend to favor different criteria or mixes thereof. A seemingly legitimate hypothesis is that as complexity grows, the need for multiple indicators grows exponentially. Our initial concern will be quantitative factors, later turning to qualitative ones. Numerical measurements have the advantage of direct comparability. When in ratio (rather than absolute) form, they also have the merit of being “objective,” that is, equally applicable to all companies. INTOPIA as an Example The International Operations Simulation/Mark 2000 (INTOPIA), representative of the class of complex integrated strategy exercises, will be used for illustrative purposes. Table 1 shows a number of activity indicators for the final period (8) of the first INTOPIA run at the American Graduate School of International Management (Thunderbird), Fall 1994. The word “activity” is used as presumably being more than comprehensive “performance.” (The difference between the two is no doubt subjective.) Individual indicators used are defined in Figure 1. For single- minded instructors, we recommend ROl, as this traditional performance measure is still widely used. Too, its advantages as well as limitations are well understood. 12 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 13 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 A few comments on the run as well as the data in the table seem desirable. Co. 1, run as monitor by the facilitator, represented a major Japanese competitor (Japan is not an operating area for participant teams). Hence, its data are excluded from the Industry Averages. Co. 4 was selling their entire output to Co. 5, which was a wholesaler. In this particular run, Co. 5 was the only company selling chips (product X) to the consumer market, for upgrading or replacement purposes. Cos. 2 and 8 were both PC manufacturers, marketing all their output, or a significant part thereof, to the consumer market. Co. 6 was vertically integrated. A glance at the cluster of profitability ratios 14 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 indicates a relatively low correlation between individual companies, with the exception of component suppliers Cos. 3 and 7. These companies were in patent pools together. It is possible that their cooperation extended to pricing and other areas, although the facilitator was not aware of such behavior, nor were there any complaints from PC makers 2 and 8. The contrast on performance variables between the latter companies is striking. Although Co. 4 evidenced the highest ROE its operations were somewhat unexciting. Despite its positive Gross Margin, Co. 6 is the only industry member running an unprofitable operation. Its great investment intensity provides a prime clue to this state of affairs (while also being a consequence thereof). Incidentally, the model does not discriminate against vertical integration. Even superficial comparison of the disparate numbers mow handful of profitability indicators strongly suggests that similar discrepancies occur in annual reports of publicly traded companies. Our criteria do not purport to be complete. However, they may be taken as applicable to any business. Our exercise above serves to emphasize that no single financial criterion is indisputable as representing company performance in complex environments. Action Potential May Well Be Number One Philosophically, we believe that business is in business to stay in business, i.e. the modern corporation is not in business to earn profits; it earns profits to stay in business. This, in turn, means that action potential for the future overrides any single financial criterion in importance. While in part reflecting performance, the variables in the four categories below Profitability Ratios in Table I are primarily indicative of action potential for the future. Again, no claim is made for a complete set. By way of example, such a set would also include an appropriate size of inventory of marketable goods, and the presence of networks, each favorable to all its members, i.e. viable standing supplier contracts, inter-company loans, patent pools, currency hedges, joint ventures. (All such standing arrangements are represented in INTOPIA.) COMPANY-SPECIFIC EVALUATION Our remarks on action potential for the future is an example of how qualitative elements inescapably will blend with quantitative factors in any balanced approach to simulation exercises, Frequently, qualitative elements call for qualitative evaluation criteria. The challenge here is to keep subjective elements of evaluation within bounds. This will be illustrated in our discussion of evaluating companies on their own merits. Assumption: Each Company Has a Right to Be Evaluated on Its Own Merits This assumption is a philosophical one, based on the notion that as individuals have a right to be evaluated not only on general criteria (such as age, IQ) but also on their own merits (degree to which their IQ, dreams, etc. have been realized), so do organizations. In other words, this is a separate basis of evaluation, distinct from the conventional general-criteria basis discussed earlier. It should be stated from the outset that this type of evaluations is a matter of personal style, practiced by the author. Such an “extra” evaluation is not a necessary part of INTOPIA operations-indeed, many users may well disagree with the approach on philosophical (or practical) grounds. Entrepreneurial Decisions The yardstick needed for evaluating a company on its own terms is provided by the concept of the business, its objectives, and plans. To the author, the most crucial set of entrepreneurial decisions is those that define the nature (idea, concept, scope, domain) of the business. The exhaustive set of such dimensions consists of product (or service) made or sold, functions performed, clientele(s) served, territorial extension, and time. The set of opportunities available in INTOPIA is indicated in Figure 2. 15 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 Co.A may be a maker of chips in the U.S. and of PCs in the EU, and a wholesaler of both products in Brazil, while serving PC makers with chips in the U.S. and EU, and marketing PCs to end consumers in EU through a captive sales organization. Chip making in the U.S. may begin in Quarter (Q) 2, PC manufacture in EU in Q3, and wholesaling in Brazil by Q4. As the illustration suggests, scores of business profiles are available from the beginning right through to the end of the simulation. Typically, companies have to select their (initial) niche within the default value of a uniform Swiss Francs 20 million starting capital (presumably supplied by venture capitalists). However, a facilitator wishing to expand the overall scope of entrepreneurial decision- making can do this by requiring companies to submit formal prospecti, and allocating starting capital according to their relative merit. The concept of the business may be accompanied by a statement of business philosophy (growth should be financed internally, stability vs. riskmindedness, innovation is our keynote, outsourcing of chips, etc.). An indispensable part of yardstick design is a statement of objectives, including quantification and schedule of major goals (Retained Earnings should be positive byQ5, regular dividend payments from Q6; by Q4 Co. should have at least 20 percent of the global PC market by revenue and by Q8 at least 25 percent, etc.). Clearly, the definition of objectives is ultimately a subjective matter-as is the business idea itself. However, various dimensions of objectives may nevertheless be objectively analyzed to a fair extent, as indicated by Figure 3 (taken from a handout to participants). Strategy Implementation: Plans Implementation dynamics is a key feature distinguishing games from cases. A couple of quarters into the simulation, companies are assigned the preparation of their business plans for Q4-7. The criteria for the evaluation of plans are, in part, common with those used for objectives. However, the following criteria are added: • Logic of ends and means • Completeness • Balance of short and long-term considerations • Assumptions 16 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 The evaluation of the company on its own merits is accomplished by laying the yardstick defined by its concept of the business, objectives and plans along its performance as evidenced by its own outputs and, to an appropriate extent, its adaptation to environmental events (such as strikes, trade regulation) and industry developments. Deviations (positive and negative) &e noted by the facilitator, and will be further examined in the management audit of the company. Expected Performance vs. Actual A comparative analysis of expected performance vs. actual may clearly be undertaken at almost any level of decision-making. This theme has been developed in detail by Richard Teach in his article on forecasting accuracy as a performance measure in business simulation (Teach., 1993). In INTOPIA, the ancillary CASH+ forecasting program provides a standard means of establishing expected performance at various levels of detail, such as sales by product, region, or at the global level, or cash flow forecasting on a regional, currency, and/or consolidated level. Actual performance is displayed by the quarterly company outputs. Positive and negative deviations may be evaluated in conventional terms (were the deviations due to managerial action/inaction, or to “unforeseeable” environmental events, etc.). Inter-Company Audit (or Self-Review) The last major assignment in my runs is a cross- company management audit focused on performance evaluation. It will typically take each company a couple of hours of preparation time. This time may be cut in half (or more) by each company undertaking a self-review, based on the self-defined yardstick and actual performance. Class discussion and evaluation of the quality of the presentation follows in either case. The minimum checklist of items to be considered includes the objective-plans-implementation sequence, networking with other companies, reaction to world events, comparative analysis focused on the closest competitors, evaluation of action potential, and, by way of conclusion, recommendations to a successor team of managers. Beyond the documentation of the focal company, sources available include consolidated financial statements for all companies for Q4 and Q8 (corresponding to annual reports), a listing of all standing contacts, current and back issues of the quarterly trade association journal, an Investor’s Almanac summarizing the objectives of all companies separately, and the overview of industry profitability ratios and activity indicators (Table 1). Naturally, good audits/self-reviews may also be performed without these accoutrements. The author has found it quite helpful to enlist the cooperation of INTOPIA participants in cross- company evaluation. The introductory part of the instrument is shown in Figure 4. The amounts invested are summed up for each company, and the companies are ranked by the “investments” received. Although it may be taken for granted that the evaluation criteria of the instructor are somewhat different, the rankings given by participants are generally (not always!) quite similar to those of the author. EVALUATION OF INDIVIDUAL EXECUTIVES Evaluation of individual participants is a different matter than the measurement of company performance. It is, however, closely related, and is often considered a vexing problem. The author 17 Developments In Business Simulation & Experiential Learning, Volume 24, 1997 uses a trifocal approach. First, early in the run participants take a test in INTOPIA “rules. This is justified, as executives in everyday life have to “learn the ropes" of any new job. Second, at midgame as well as endgame, the members of each executive team are asked to fill in a Group Analysis Form, evaluating other members of the team (and themselves) on 10 different criteria, using a scale of 1 to 10, or 1 to 20 on each. To minimize collusion and camaraderie effects, participants are told that forms with identical rankings on any single criterion will be disregarded if such identity occurs more than twice. Participants are also reminded that as practicing executives they will be asked to evaluate the performance of close colleagues and subordinates. Finally, two or three written individual assignments pertaining to planning and creativity and/or data analysis are required during a semester. Evaluation Criteria Ex Ante and Ex Post The point has been made that evaluation criteria must be specified in advance of a run. This applies to qualitative as well as quantitative elements of performance. In addition, the author has found an important part of debriefing to be a discussion with participants of alternative performance criteria. Such a discussion can provide a valuable learning experience for instructor as well as participants. REFERENCES Anderson, P.H. & Lawton, L. (1992) “The Relationship Between Financial Performance and Other Measures of Learning on a Simulation Exercise”, Simulation & Gaming, 23, 326-340. Gosenpud, J. & Washbush, J. (1995) “Simulation Performance, Learning, and Struggle,” Developments in Business Simulations and Experiential Exercises, 22, 1-4. Palij, P. (1995) International Operations Simulation/Mark 2000 Reviewed by Peter Palij. Simulation & Gaming, 26 (4) 524-527 Teach., R. (1993). Forecasting accuracy as a performance measure in business simulations. Simulation & Gaming, 24 (4), 476-490. Thorelli, H., Graves, R., & Lopez, J-C (1995) INTOPIA Executive Guide. Englewood Cliffs, NJ: Prentice-Hall Thorelli, H., & Lopez, J-C (1995) INTOPIA Compendium for the Administrator Englewood Cliffs, NJ: Prentice-Hall. 18 Table of Contents Volume 24, 1997 Incorporating Computer Telephony into the MIS Course A Learner Oriented Infrastructure for videoconferencing Based Distance Education Courses The Identification of Temporally Related Structural Elements of the Experiential Component of Electronic Spreadsheet Tasks Does Involvement Influence Learning from Simulation Participation? Some Relationships with Helpfulness and Performance Outcomes Musings on Business Garne Performance Evaluation Performance on a TE Simulation: What does it represent? The Business Policy Game: An International Simulation - An Assessment Tool Students in Free Enterprise as Experiential Learning The Role of Computer Models in Wargames: A Practitioner's view Using Focus Groups as a Tool to Research Learning: A Demonstration and a Discussion Using Medical Simulations to Teach Multi-Cultural Diversity Using the World Game to Internationalize International Exchange Game Community Services Needs Assessment: An Innovative Approach Introducing Students to Potential Total Quality Management Ethical Dilemmas Privacy in the Workplace: A Situational Analysis Antecedents of Learning in Simulations Demonstrating the Learning Effectiveness of Simulation: Where we are and Where we need to go The Use of Computer Simulations as a Pedagogical Aid in Teaching Management Information Systems Evaluating Simulation Learning in a Distant Learning Instructional Model Financial Engineering of Global Investments The Labor History Game: Playing with the Past Perks Participation Assessing Negotiator's Proficiency with a Negotiation role-play The Art of Negotiating Enhancing Learning and Employee Development through the Assessment of Learning Pedagogy Preferences across selected Dimensions of Culture: A Preliminary Investigation Cooperative Learning: What are we Learning? Effective Use of Mastery Based Experiential learning in a Project Course to improve skills in system analysis and Design Simulations and Learning: Can we prove a relationship? (Seminar) Predicting and Reviewing NYSE Stock Prices by use of Basic Statistical Analysis and Logic Toy Car Depreciation Exercise ABSEL as Home Community: An Interactive Exploration Service Learning: Linking Academic Study to Community Development and Business Enhancement Ability of Efficient Evaluation of Knowledge-Based Management Strategies Corporate Ethics Training Programs Modeling Attributes in Demand Functions of Computerized Business Simulations: An Extension of Teach's Gravity Flow Algorithm Expert Systems Combined with Neutral Networks: Tools to Benefit the Marketing Researcher Computer Game Design: New Directions for Intercultural Simulation Game Designers Consistency in Simulation Performance over Time and Across Simulation Games The Impact of an Artificial Market Leader on Simulation Competitor's Strategies Business Plans, Case Studies, and Total Enterprise Simulations: A Natural Co-existence An Exploration into the Non-Use of Business Simulations The Market Game: Interactive Learning Through Market Simulation Plotting Brand Trajectories with the COMPLETE PPM package: A Market Segmentation Analysis and Positioning Tool Me and Mine Inc: An Exercise in Management Theory Learning to Differentiate Leadership from Managerial Position Business Ethics Survey: A Perspective from the Retail Industry Experiential Learning in Demand Analysis for an Agricultural Commodity Marketing on the Internet: A Pedagogical Exercise College Students Need Simple Computer Simulations, Especially for International Business Courses An Analysis of Student Attitudes, Performance, and Strategies in a Simulation Competition Based on a Controlled Product-Market-Entry Game Structure The Crystal Enterprise: Application of a Non-Computerized Simulation Model in a Process of Organizational Change Threshold: A Windows-Based Behaviorally Oriented Total Enterprise Simulation Coaching Business Game Teams Using a Decision Variable Optimizer Rock and Roll is here to Stay: Enhancing Experiential Pedagogy with Musical Exercises An Example of Business Process Analysis Simulation for Customer Software Support The Use of Business Gaming in Hong Kong Academic Institutions Using the Integrative International Simulation INTOPIA Mark 2000 in a Concentrated MBA Curriculum A Group Experience as an Integrated Part of the Core Management Course Empowered Learning in the Classroom The Inter-Group Interaction: An Innovative Approach to Cooperative Learning Courses that Utilize Student Team: An Approach to enhancing their Effectiveness Thoughts about the Measurement of Learning: The Case for Guided Learning and Associated Measurement Issues Measuring Student Learning Using Business Simulations: A Theory Based Perspective On the Use of PC Fingame in an Undergraduate Finance Course How Managers get Things Done: A Virtual Soundbite Internet Experiential learning in the Principals of Marketing Classroom: A Pedagogical Approach Current Student Perceptions Relative to Business Simulations Beyond Capitalism: Designing Business Policy and Social Justice The Use of Boards of Directors to Evaluate Reports and Presentations in an Undergraduate Business Policy Course Beacon Lumber: An Experiential Introduction to Financial Accounting The Application of Organizational Motivation Principles: The Experiential Business Simulation Motus Manufacturing Avoiding a Bogey: Grading Case Discussions Scientifically Communicating Consumer Behavior II: A Modified Exercise Using Personal Consumption Journals in Condensed Courses Designing Instruments for Assessing the Effectiveness of Simulations, (Seminar) 360º Performance Feedback: Appraisal vs. Assessment The Cafeteria Approach to Managing an Academic Career: Remaining Non-Perishable while Doing Your Own Thing An Experiential Exercise Related to Person-Organization Fit and it's Consequences For Today's Dynamic and Changing World, Business Simulations need to be expanded to encompass Much Greater Complexity: A Demonstration The Incident Process: A Case in Reverse Contextually-Anchored Business Simulations The Energy Factor: Building Motivation in the Simulation Gaming Environment The Design of an Internet Game