THE AMBITION GRADIENT APPROACH TO EVALUATION OF COMPUTER SIMULATION GAME TEAM PERFORMANCE Developments In Business Simulation & Experiential Exercises, Volume 20, 1993 19 THE AMBITION GRADIENT APPROACH TO EVALUATION OF COMPUTER SIMULATION GAME TEAM PERFORMANCE Alvin C. Burns, Louisiana State University ABSTRACT Evaluation of the quantitative performance of company-teams is difficult due to their uneven performance profiles and different strategic postures. Proposed here is a gradient evaluation system, which embodies seven different propositions on business simulation team performance evaluation. The ambition gradients approach derives a percentage corresponding to how well a company-team performs relative to its goals. Effective and efficient goat gradients comprise a diagnostic grid, which also reveals performance relative to other company-teams. While not the perfect evaluation method, the ambition gradient approach does allow direct comparison of disparate strategies, and it can be used to track the learning/performance of a company-team over time. INTRODUCTION Computer simulation competition is touted to be an excellent form of experiential learning. However, the evaluation of the performance of a company managed by a student team competing against other company- teams in sustained simulated game play is arguably the most difficult task confronted by the game administrator. The difficulty of this responsibility is exacerbated by theoretical debates on what constitutes experiential learning. For instance, one argument holds that in order for learning to occur, some form of failure must first result (Gentry, 1990). Another camp contends that trial and error, possibly random in nature, is necessary for learning (Gentry, 1990). Still another view is that improved performance must be clearly evidenced. In addition to the academic debates swirling around this topic, there are numerous practical problems posed by the typical uneven performance profiles exhibited across teams. Although the use of multiple indicants is customary, (see for example, Wolfe and Box, 1988, Miesing, 1982, Anderson and Lawton, 1988), for the purpose of illustration, let’s assume that profit is the only indicant of performance. Consider the following types of unevenness. First, a company-team’s profit may be inconsistent over the duration of the game. Most games commence with equal positions, but over game play, a team may dip down initially then exhibit a slow ascent to be the leader. Alternatively, it might zoom to the leader position, then slump to be a distant follower of the ultimate leader. Still another scenario is the consistent second- or Third-place Company. On a different level of unevenness, it is conceivable that a company-team will opt for a niching or market segment specialist strategy, which is acceptable from a strategic implementation standpoint, but which also places the company-team far from others on the performance measure. These examples illustrate the types of problems faced by simulation game administrators faced with the evaluation of computer simulation company- teams. This paper addresses these problems and describes an evaluation mechanism which may prove useful to team administrators. It begins by noting some basic issues involved with the company-team evaluation task and posits a set of propositions, which should underpin an evaluation system. Next, it introduces the concept of “ambition gradients” and describes their computations, interpretations, and diagnostic implications. Last, to show the application of the ambition gradient method, a desirable performance profile over time is illustrated. PERFORMANCE INDICANTS FOR COMPUTER SIMULATION TEAM PLAY The focus here is on the quantitative performance of a company-team engaged in sustained computer simulation play and pitted against other company-teams. Granted, there is a myriad of team dynamics, strategic planning, and concept understanding and applications performance attributes which can be monitored. We readily acknowledge that these factors are important in varying degrees depending on the course level, instructor’s orientation, course content, and nature of the game, but they are not the focus of this paper. This paper is delimited to the question of what quantitative performance measures are appropriate and how should they be used in a fair comparative sense. Typically, a computer simulation issues a wide range of output information. Sales volume, sales revenues, market share, inventory, return on investment, and a host of financial ratios are standard, and each one is useful to company-teams making decisions in some specific way. As performance measures, however, all cannot and should not be used. Some are highly correlated. For example, sales volume in units and market share must be highly correlated because volume in units is in the numerator of the market share formula. Others are not correlated, but they are useful primarily as signals of relatively minor decision errors. For example, inventory shortages may simply highlight the improper use of an economic order quantity model. The task, then is to select those performance measures which reflect the goodness” of the company-team’s overall business decisions, but these measures should at the same time accommodate the variety of uneven, yet normal movements and unique strategic posturing of company-teams. The point to be made us that multiple performance indicants are needed in order to satisfactorily evaluate the progress of a particular company-team. In the interest of parsimony, we will focus on three generic performance constructs: (1) effectiveness, (2) efficiency, (3) relativeness. Effectiveness pertains to how well the company-team has achieved its sales goals. This construct is consistent with Anderson and Lawton (1990, 1988) who argue that the ability of a company-team to realize the predicted results of its decisions is one performance criterion. Efficiency refers to how well it has managed its marketing and other cost factors in achieving whatever level of effectiveness it has gained. This criterion is an indicant of the company- team’s financial health, and it is consistent with many authors who have noted the use of profit, rate or return, or stock price as a performance measure (See for example, Miesing, 1982, or Wolfe and Box, 1990). Finally, relativeness alludes to the effectiveness-efficiency posture of the company-team with respect to its competitors. Again, Anderson and Lawton (1988, 1990) have observed that relative position is a common form of performance evaluation. Separate from what indicants to use is the issue of how to use them. One alternative is to focus only on end-of-game performance. Here, the positions of company-teams are sometimes compared against some absolute standard, or, more likely, they are compared against each other in a relative standing sense. An optional system is to evaluate change in performance, comparing end-of-game position in profitability, for instance, with the company-team’s situation at mid-game. Finally, there is strategic performance analysis where the company-team’s position is subjectively evaluated vis-a-vis its strategic orientations across the game. Again, advocates can be found for each evaluation system, as well as for other systems. PROPOSITIONS ON EVALUATION OF SIMULATION TEAM PERFORMANCE As can be seen, the evaluation problem has many facets, and each one encompasses weighty issues. As a first attempt at attacking these issues, this section of the paper specifies several propositions, which we would argue should underpin any evaluation scheme. In any case, they underlie the gradients approach evaluation system later described. Each proposition will be stated and briefly explained. P1: The evaluation system should be multifaceted. While the focus Developments In Business Simulation & Experiential Exercises, Volume 20, 1993 20 here is solely quantitative company-team performance, the argument for the use of multiple measures seems well grounded. We noted earlier the plethora of information normally generated by a computer simulation and provided as output. Evidence of this is apparent in Burns and Gentry (1992) who identify several financial and other performance data provided to students and instructor with each game period. P2: The evaluation system should be consistent across company-teams. Consistency is mandated by the fact that a learning process is being evaluated, and students should be accorded fairness in the evaluation system. In theory, the evaluation system should share the consistency properties of objective examinations. Subjective evaluation schemes are inherently biased. Moreover, knowledge that a subjective evaluation will take place renders company-teams indecisive as to how to operate. P3: The evaluation system should be geared to company-team objectives. As was noted earlier, company-teams pitted against one another in mid- game play will exhibit a wide range of objectives in any given decision period. Objectives differ by company-team due to period-by-period shifts in position. Consequently, an evaluation system must take these differences into consideration. P4: The evaluation system should be adjusted for strategic Orientation differences between company-teams. Business/marketing strategy theory allows for diverse yet successful strategic orientations. As an example, Kotler (1991) describes four separate marketing strategy orientations of market leader, follower, challenger, and nicher. He claims that under given circumstances, each one is appropriate and profitable. The myriad of strategic orientation possibilities is perhaps the single most troublesome aspect of evaluating the performance of company-teams. Ignoring differential strategic orientation flies in the face of business/marketing strategy theory and practice. P5: The evaluation system should take into account the effectiveness and efficiency of company-team performance. Conceptually, effectiveness and efficiency are independent performance dimensions. That is, a company- team may dominate an industry in market share (effectiveness) but be the least profitable player (efficiency) at the same time. Similarly, high profitability may characterize a company-team with low sales volume. An artificiality of business simulations intensifies the need to measure effectiveness and efficiency simultaneously, for company-teams cannot stop playing even when they find themselves in financial straits which would destroy real-world companies. P6: The evaluation system should take into account relativeness. An implicit tenet of all competitive games is winning, yet only one company- team can be ranked number one. Since advancement in ranking (or prevention of slippage) is a driving force, the relative performance of company-teams needs to be included in any evaluation scheme. The problem is to find a relativeness measure, which takes into consideration the previous five propositions as well. P7: The evaluation system should be diagnostically meaningful and useful. This proposition holds that an evaluation system should have the ability of being applied throughout the game and be interpretable to company-teams for managerial implications. This proposition is admittedly bold, for it calls for a system that can be used by company-teams to recognize errors and to govern their actions during game play based on this error recognition system, that is, to facilitate learning. THE AMBITION GRADIENTS APPROACH Again for the purposes of illustration, we will assume that the effectiveness of a company-team is reflected by its market share, while its efficiency is measured by the company-team’s per unit profitability. These two measures are logically unrelated, as it is possible to have large market share with any level of per unit profitability ranging from high positive to high negative. Similarly, a small market share company-team may experience profitability along the same continuum. The Ambition Gradient Formula Before providing the formula, we should define the notion of ambition.” Cook (1 983, 1985) borrowed the idea of competitive ambition from military tactical theory and introduced it to the marketing field. However, our formula differs significantly from Cook’s calibration. With the Cook approach, marketing ambition is the amount of marketing (e.g., width of product assortment, amount spend on promotion, number of outlets, etc.) devoted to achieving the firm’s objectives. Thus, for Cook, ambition applies the principle of force (Cook, 1983). The more the force, the higher the ambition. Our ambition gradient is more consistent with Anderson and Lawton’s (1990, 1988) recommendations that the company- team’s predictions, operationalized here as its objectives, be used in the evaluation scheme. A gradient is a measure of the change in two factors expressed as a ratio or a percentage. Thus, the effectiveness and the efficiency ambition gradients are both calculated with the same formula, which is: As can be seen, the gradient concept compares the change in performance which actually occurs to the coal or desired change (ambition). In other words, if a company-team sets a goat of a 10% increase in market share in coming decision period, and it realizes only a 5% market share increase, it has experienced a 50% effectiveness ambition gradient. Similarly, if its per unit profitability goal is a 20% increase, and the income statement reports a profit increase of 30%, the efficiency ambition gradient is 150%. Graphical Presentation of the Ambition Gradient Figure 1 presents the ambition gradient concept graphically, and it illustrates some desirable properties of the gradient. In the Figure, the X- axis is identified as the Actual Period Change, or the percentage of the company-team’s beginning-of-period position realized at the end-of-period. In other words, if a company-team had 40% market share at the beginning of the period and effected a 50% market share from its decisions, the Actual Period Change would be (50%/40%) or 125% increase. This axis is the barometer of change against which company-team goal fulfillment is measured. The Y-axis is labeled Ambition Gradient Value, - and it is the value determined from the gradient formula. Notice that the Ambition Gradient Value is expressed in percentage, and its interpretation is directly interpretable in terms of the company-team goal, that is, ambition, as will be explained next. Figure 1 illustrates the ambition gradients for three different goals: (1) a 30% increase, (2) a 20% increase, and (3) a 10% increase. The different slopes are a direct function of the goals, yet despite different goals, the gradients are directly comparable. Here are the comparable aspects. First, when the gradient is zero, all gradients intercept. That is, when the end-of- period performance is no different from the beginning-of-period level, zero percent of the goal has been attained. Second, when the gradient is 100%, the goal has been attained exactly. Note in Figure 1 that the 100% gradient value cuts the gradients at 110% on the 10% gradient. 120% on the 20% gradient, and 130% on the 30% gradient. Third, when the gradient values are compared on any given actual percent change, they are interpretable as percent attainment. For instance, at the 1 20% Actual Period Developments In Business Simulation & Experiential Exercises, Volume 20, 1993 21 Change, the gradient is 200% indicating a double 10% goal actual performance, 100% of the 20% goal attainment, and 67% of the 30% goal. Fourth, the gradients are interpretable in a symmetric manner. That is, when the gradient is -100%, it means that not only did the actual decrease, but it decreased at precisely the amount of increase specified as the goal. An especially valuable feature of ambition gradients is their accommodation of strategic orientations. To illustrate how the effectiveness gradient handles disparate strategic orientations, we will take the case of a company-team intentionally downsizing its market share which would occur with a niching strategy orientation. Suppose the goal was a 10% decrease in market share. Figure 2 illustrates the -10% effectiveness ambition gradient. Notice that it has the opposite form of a 10% increase gradient, which we have included on the graph for comparison. That is, if the company-team reached its goal of a 10% decrease, effecting a 90% level of the beginning market share, its gradient would be 100%. If it failed in its objective, say for example, that its market share actually grew, its gradient would be negative. So any intentional downsizing objective is handled by the ambition gradient approach. USE OF AMBITION GRADIENTS TO EVALUATE COMPANY- TEAMS A Diagnostic Grid for the Ambition Gradients It is important to reiterate that the gradient concept applies to efficiency as well. That is, profitability ambition is treated identically. This allows for an effectiveness-efficiency gradient matrix as a means of diagnosing the period-specific change of a company-team’s position. Figure 3 illustrates the possible use of this tool. In Figure 3, we have identified three levels of gradient performance: (1) low, (2) acceptable, and (3) high. The result is 9 different cells, only one of which is “acceptable.” The use of this grid is diagnostic in that a company-team can see where effectiveness has been gained at poor efficiency, or where efficiency has resulted in ineffectiveness. The diagnostic grid raises the question of “What is acceptable?" The answer to this question is complex, but one way to approach it is to refer to the sixth proposition noted earlier: The evaluation system should take into account relativeness. With several periods of game play, or, alternatively, if several products are marketed by each company-team, and each item has ambition aspects, average deviations above and below the 100% gradient level can be calculated. Thus, acceptable performance would be that which falls in the average range; low performance would be that which fell below acceptable, and high would be that which fell above acceptable. The notion of a low ambition gradient is intuitive, but a high ambition gradient bears explanation. That is, with a low gradient, the company- team’s performance has failed to reach its goal, which is a common occurrence. But with a high gradient, the actual performance is greater than the goal. This condition means that the predictive ability of the company- team, that is, its goal setting, is faulty. To word this differently, it has erred in an important decision making skill. This skill must be improved in order to return to acceptability. Over time, a company-team should exhibit a pattern of “moving to acceptability. - However, it is important to note that acceptability is also moving. That is, if all teams improve, then the average deviations from 100% for effectiveness and efficiency will also narrow. In our experience, the acceptability range always remains reasonable for three reasons. First, randomness in a computer simulation game will insure that the ambition gradients fall above and below 100% across company-teams in any given period of play. Second, since an average of the ambition gradients for a given period is used, about one-half of the company-teams will typically fall into the acceptable range in any given period. Last, learning slopes of company-teams vary, so some are more likely to generate unacceptable ambition gradients than are others. Ultimately, however, the average range approach should be replaced by some arbitrary level such the range of - 110% to + 110% as a means of affording all teams the opportunity to attain acceptability. Figure 4 illustrates what we mean by movement to acceptability. A company-team will exhibit high/low efficiency/effectiveness, but as its ambition and actual performance converge due to learning, its ambition gradients will fall into the acceptable range with some consistency. If they do not, learning has not been demonstrated. The diagnostic aspects of the ambition gradient approach underlie learning. When the gradient level is low, ambition (goal) is inconsistent with reality. Here, either the effort (price, promotion, etc.) must be altered, or the goal must be lowered. When the gradient level is high, ambition is also inconsistent, but here, the goal should be raised, or effort reduced. The ambition gradient approach is certainly no panacea to the complex problem of evaluating company-team performance, but it does appear to be a useful tool in that it overcomes many of the problems accompanying single-indicant performance measures such as end-of-game total profitability. It embodies the seven propositions listed earlier, and it Developments In Business Simulation & Experiential Exercises, Volume 20, 1993 22 applies constant pressure on company-teams to improve across at least three dimensions: effectiveness, efficiency, and relativeness. Unfortunately, use of ambition gradients will not automatically score company-teams and assign them simulation game play grades. Ambition gradients will, however, afford the game administrator a dynamic, graphical, and reasonably comprehensive evaluation tool. REFERENCES Anderson, Philip H. and Leigh Lawton (1988), Assessing Student Performance on a Business Simulation Exercise, Developments in Business Simulation and Experiential Exercises, Vol. 15, Patricia Sanders and Tom Pray (eds.), 241 -245. Anderson, Philip H. and Leigh Lawton (1990), “Methods for Evaluation Performance on Business Simulations: A Survey, - Developments in Business Simulation and Experiential Exercises, Vol. 1 5, John Wingender and Walt Wheatley (eds.), 177. Burns, Alvin C. and James Gentry (1992), “Computer Simulation Games: Past, Present and Future,” Marketing Education Review, Vol. 2, No. 1, Summer, 3-13. Cook, Jr., Victor J. (1983), “Marketing Strategy and Differential Advantage," Journal of Marketing, Vol. 47, No.2, Spring, 68-75. Cook, Jr., Victor J. (1985), “Understanding Marketing Strategy and Differential Advantage, - Journal of Marketing, Vol. 49, No. 2, Spring, 137-142. Gentry, James (1990), “What is Experiential Learning,” Guide to Business Gaming and Experiential Learning, Nichols/GP Publishing, 9-20. Kotler, Philip (1991), Chapter 14, Marketing Management; Analysis, Planning Implementation and Control, Prentice Hall, Englewood Cliffs, NJ, 375-387. Miesing, Paul (1982), “Qualitative Determinants of Team Performance in a Simulation Game, Developments in Business Simulation and Experiential Exercises, Vol. 9, David J. Fritzsche and Lee A. Graf, (eds.), 228-231. Wolfe, Joseph and Thomas M. Box (1988), Team Cohesion on Business Game Performance, - Simulation and Games, Vol. 19, No. 1, March, 82-98. Table of Contents Volume 20, 1993 Dominant Personality Types and Total Enterprise Simulation Performance Shelf Wars: A Grocery Channel Simulation Shared Cultural Perspectives: An Experiential Exercise Utilizing International Students to Globalize the Classroom An Instrument for Investigating the Effectiveness of Teaching Methods in the Business Policy and Strategy Formulation Course Providing Better Trained Graduates for Accounting Employers The Ambition Gradient Approach to Evaluation of Computer Simulation Game Team Performance Alphatec: A Negotiation Exercise with Logrolling and Bridging Potential Using the Ideafisher Idea Generation System as a Decision Support System in Marketing Strategy Courses A Dynamic Market Share Allocation Model For Computerized Business Simulations Multi-Cultural Adaptability Using Experiential Learning in a Graduate Course Development of Experiential Applications in HRM: Practicing What Preach and Preaching for Practice Linking Students and Business Leaders Through Portfolios Debriefing International Experiential Learning Exercises: Road Signs for Effectiveness Sales Manager: A Simulation Modeling Interactive Effects in Mathematical Functions for Business Simulations: A Critique of Goosen's Interpolation Reducing the Complexity of Interactive Variable Modeling in Business Simulations Through Interpolation Antecedent Biases of Experiential Learners: Trainee Occupation and Subgroup Diversity Pax in Terra Sancta: Simulating the Middle East Peace Negotiations A Multiple Regression Case In Experiential Learning Changes in Ethnocentric/Geocentric Orientation by Business Students after Exposure to a One Summer Course in International Marketing's A Systematic Approach to the Development and Evaluation of Experiential Exercises Entrepreneurs Evaluate Experiential Education A Linear Programming Approach to Open System Total Enterprise Simulations Reflecting Leader Behavior from the Looking Glass, Inc. Simulation Linking Cognitive Styles, Teaching Methods, Educational Objectives and Assessment: A Decision Tree Approach Restructuring Management Education in Post-Communist Countries: How Western Experts Can Help Managerial and Cultural Pre-Conditions for Superior Performance in a Global Setting: An Experimental Study with the Aid of Business Games Multiple Industries in Computerized Business Gaming Simulations Content or Process? - Content and Process! Some Observations and Reflections About Management Education in Central Europe Out-of-Class Experiences to Promote Volunteerism Enacting the Linguistic Consciousness of the Modern Managerial Mind: Post-Modernism and Experiential Learning Intergrating Experiential Exercises into the College Curriculum: The Case of Internationalizing the Business Curriculum Simulation Marketing Oversights Incorporating Advertising Creative Strategy into Computer-Based Business Simulations The Dynamics of a Partnership Between Business and Education Collaborative Education Done Globally Experiential Systems Analysis CADPLAN: A Simulation for Comparative Advertising A Doctoral Symposium: Preparing Students for Conference Behavior Comparing the Simulation with the Case Approach: Again! Total Quality Management: A Model for Continuous Quality Improvement The Quality Audit: An Experiential Exercise for Business Students Extending the Reach of Simulations: DECIDE Heads for the Inner City Lessons Learned from a Customized Management Development Simulation The Foreign Exchange Spot Trading Simulation Using Lotus 1-2-3 to complete a Triple Play in a Simulated Competition International Business Education: Is Enough Being Done? Matching of Student-Teacher Cognitive Style as a Factor in Student Success in an Introduction to Information Systems Course Breathing (More) Life into the Case Approach Lord of the Flies: A Live Case Approach to Leadership Cooperative Case Studies: Experiential Tools for Teaching Business Problem Solving Tools Strategy Simulations in Context: An Evaluation of Key Dimensions The Distribution Channel Game Evaluation of a Simulation Game as an Education Tool for Utility Professionals The Relationship Between Total Enterprise Simulation Performance and Learning Total Quality Management does not Happen by Magic, but it can be Taught Using a Pedagogical Methodology that Utilizes Magic Effectively Preparing Students for Careers in a Global Environment by Integrating Total Quality Management Thoughout the Business Curriculum An Empirical Investigation of Cognitive and Performance Consistency in a Marketing Simulation Game Environment Using MARSGAP with LAPTOP: (A Marketing Simulation Game Analysis Program) with LAPTOP: A Marketing Simulation Adapting TQM Implementation to Organizational Level An MBA Business Simulation: Executive Interaction Experiential Exercises and Pedagogy Track Workshop: Experiencing Cultural Diversity in the Classroom (and the Hotel Meeting Room) Closing the Gap between Corporate and National Culture The Dynamic Manufacturing Company The Use of Experiential Techniques in Corporate Training The State of Simulation Gaming in Easter European Countries- Principally Russia An Experiential Exercise in Cross-Cultural Training Valuing Differences: A Conceptual Framework Demonstration of an Experiential Exercise Effectively Using Experiential Learning to Impart TQM Concepts in a High Technology Environment The Older Worker Questionnaire: An Exercise Concerning Older Worker Stereotypes and Behaviors The Crime Fighting Task Force: An Exercise in Organizational Politics Welcome to the Party! An Expression of Vocational Preference Experiential Exercise for Imparting Cross-Cultural Appreciation Six Swift Simulations on Globalization Overview of BASF Delegate Program