EVALUATION OF PERFORMANCE IN MANAGEMENT SIMULATION: A MANAGEMENT COEFFICIENTS MODEL Developments In Business Simulation & Experiential Exercises, Volume 22, 1995. 136 EVALUATION OF PERFORMANCE IN MANAGEMENT SIMULATION: A MANAGEMENT COEFFICIENTS MODEL Sharma N. Pillutla, Towson State University Lloyd A. Swanson, Syracuse University ABSTRACT This paper discusses the grading issue for a simulation game being played for the introductory operations management class. Evaluating performance of the game participants has been an issue of great interest and research amongst game administrators. Ultimately, however, whatever means of evaluation are used, the rating has to be turned into a class grade for the participant. For multiple sections playing the same game, an important issue is maintaining consistency in grading. This paper looks at one approach based on Bowman’s [1] management coefficient approach that alleviates the consistency problem and automates the grading procedure. INTRODUCTION Management simulations as a learning vehicle have increasingly popular in recent years, especially in schools of business. Usually some proportion of a course’s total grade is based on the team’s or individual’s performance in the simulation exercise. In this paper we report an application of the management coefficients model of Bowman [1] to the problem of evaluating student performance on a simulation game. Frequently the success or failure of management simulation games in the classroom environment is dependent upon proper and fair simulation grading. Quite often the objectives given the participants are to maximize profits in the long run. However, this has proved to be insufficient. There has been some difficulty in deriving a single measure and several have indicated a preference for multiple measures of performance. For instance, Gray [6] purports the use of a linear combination of several measures, such as market share, profit performance, inventory performance, etc. to obtain a single such measure. Hand and Sims [7] utilize the technique of path analysis to investigate relationships among thirteen performance criteria. They were able to reduce the number of performance criteria down to two, viz., sales forecasting error and profit. Multiple performance criteria rankings can exert a positive influence on students by providing feedback necessary for improving several areas of decision making. However caution should be used against mechanically using these rankings in assigning grades. One of the principal problems of the game administrator is not only to identify the appropriate criteria, but also to assign a proper set of weights for the different criteria in computing an overall performance grade. Additionally, there is also the problem of specifying the proper form of weights, i.e. should it be linear or of some non-linear form. Then too there is the random element, such as sales variability, that may very well influence results and should not be ignored. The severity and nature of the competition again may play a significant role in the valuation process. Should cardinal or ordinal values rankings be the principal measurement devices? These problems have no easy answers. However, an equitable and realistic performance measure for grading purposes is of considerable importance for successful educational value. A MANAGEMENT COEFFICIENTS APPROACH One possible approach to obtaining such a performance measure is to identify all the variables, determine their relationships, find a suitable measurement scale, and assign suitable weights to each. This may be possible but is clearly a very costly effort without any guarantees of success. Another approach is to view the evaluation process more as art input-output model where the various criteria and environmental measurements act as inputs and overall simulation performance grade acts as the form of the output. Thus the former approach attempts to determine and justify the form of the relationships, the weighting of various criteria, and the interaction between the various environmental and performance measures. In the latter approach, the game administrator’s actual past decisions are used to determine the relationships between the inputs and the output, viz., the overall simulation performance grade. This approach is certainly more pragmatic than utopian in that it starts with the administrator’s actual decisions and builds a better system by eliminating the variability in those decisions. This approach is certainly not new. Bowman [1] built such a model to solve the aggregate production and employment scheduling problem. Other approaches available at the time required a rather extensive cost collection and development of a rather elaborate mathematical model. On the other hand Bowman’s approach only employed multiple regression analysis to relate the data available to the manager at the time the decision was made to the actual made at that time. Paramount to developing and utilizing such an approach are two key assumptions: that the form of the cost structure is ‘U or “dish shaped”, and that the manager has not been consistently ‘off- center” or consistently biased. Thus, although there may be variations in decisions, the assumptions imply that experienced managers are aware of the criteria and the variables that influence the system and act accordingly. Such an approach may be appropriate in many management simulation game situations to aid in determining the grade in a classroom environment. It would be appropriate where the game administrator was very familiar with the simulation and various criteria employed, and had also gained some experience in making grades in this environment. It is also basic that the criteria be quantifiable. Thus, a management coefficient model could be developed where the various performance measures would serve as independent variables and the past grading decisions would be the dependent variables. There are several advantages to such a method. The data for constructing the model is readily available from the simulation results. Secondly, the results can be easily computerized by simply adding a few statements to the simulation game program to print the final grade. Also, employing a coefficients model not only reduces variation in assigning grades within a class but also from class to class and semester to semester. In large multi-section courses such features would be most welcome. Developments In Business Simulation & Experiential Exercises, Volume 22, 1995. 137 AN APPLICATION OF THE MANAGEMENT COEFFICIENTS APPROACH FOR GRADING To demonstrate the applicability of employing a management coefficients model to assist in grading student simulation performance, the approach was applied to a simulation game utilized in an introductory Operations management course. In this game evaluation points are awarded for such categories as market share, inventory management, cash management, profit performance, etc. The relationship between the evaluation points and the performance on each of these categories has been subjectively determined. Although the weights - both slopes and intercepts - for each of the criteria were initially subjectively assigned, the weights were altered over a period of several years in an effort to identify key managerial areas and areas of concern. The points for each category as well as the sum total are reported back to the participant at the end of each decision period in an effort to identify key managerial areas of concern. While the current weighting is satisfactory, there still exists some discomfort in converting the sum of the reported evaluation points into a meaningful grade. This simulation employs three companies within independent industries with individuals making decisions for each company. Consequently there are considerable differences among the industries even though the total sales available to each industry are equal. In brief, it has been observed that some industries tend to be more competitive than others and as such two observations have been made: the more competitive industries tend to have lower average evaluation points but companies within these industries tend to do a better job in managing their companies. Since the evaluation points emphasize profits, non-competitive industries with higher profits end up with higher evaluation points despite paying little attention to other management areas. Thus there was a need felt to capture the varying degree of competition in different industries. The measure of industry competitiveness devised is elaborated below. The players in the game primarily make four decisions related to the capacity utilization, the quality of the product, the distribution effort and the price charged for the product. While the first factor labeled capacity factor (cf) addresses the supply side of the firm, the remaining three factors (the quality factor, qf, the distribution factor, df, and the price factor, pf) influence the demand for the product. In trying to compete for the market sharc, individual players either raise the qf and/or df and lower the pf in an attempt to procure higher sales for their firms. The impact of these three factors is captured in a composite factor called the Share of the Market Factor (SMF). The SMF is given by the following relationship: Higher is the SMF of a firm, the more is the effort being input in vying for market share. An average of the SMFs of the three competitors within an industry therefore gives an index of the competition within an industry. This is labeled the competitive index for an industry and represented by C. Prior to the development of the competitive index, the game administrator was forced to continually make subjective decisions on grades based on a variety of factors. This consumed a rather large proportion of time. To minimize this effort and to be more consistent in grading, a management coefficients model was constructed where measures of industry competitiveness and evaluation points served as independent variables and previous grading decisions served as the dependent variable. It was felt that there were two principal determinants in arriving at the grades: performance evaluation points and a measure of industry competitiveness. However, since the exact form of the relationship was unknown, stepwise regression was employed for variable selection. The data utilized for this analysis consisted of 83 observations from three different classes. The results of this analysis, not surprisingly, resulted in but one variable entering the solution yielding the following coefficient model: where C is the adjusted competitive index, P, the evaluation points and G, the grade points awarded. The correlation coefficient, r, was quite high, 0.753, and the F-value, 107.06, for 82 residual degrees of freedom indicated a very high significance. ANALYSIS OF RESULTS In an effort to check the validity of the model, the predicted grades were all individually compared to the actual grades given by the administrator. Wherever significant discrepancies occurred a more detailed analysis was made. The conclusion reached was that some errors were made in originally assigning grades and that the model gave more consistent grades than the administrator. It should be pointed out that the model is only appropriate for this particular simulation with a similar sales pattern and for approximately the same duration, 7 to 14 decision periods. It reflects the attitude and opinions of this game administrator. However, the model does offer greater consistency in grading and considerably reduced the effort of assigning grades. It was also found that although this model only reflects the decisions of one administrator, other instructors also were found resorting to the model for grade determination. Thus, such a model can be utilized to obtain grading decisions from people who are knowledgeable of the simulation when the instructor using the simulation is not. SUMMARY AND CONCLUSIONS In this paper a management coefficients model was developed to assist in assigning grades for management simulation games where the game administrator’s actual decisions and various simulation results were utilized in determining the regression model. The model developed in this paper, although only appropriate for the particular simulation employed, has shown to be very successful since its implementation. It should be emphasized that the model is more facilitating in nature rather than optimal as it only reflects the attitude and opinions of the game administrator. What this paper does suggest is that a similar approach can be applied to other simulations and thereby relieve one of the burdens of administering a game, viz., assigning grades. Such an approach should also result in having more consistent grading. References provided on request Table of Contents Volume 22, 1995 Simulation Performance, Learning and Struggle Are Good Simulation Performers Consistently Good? Cognitive and Behavioral Consistency in a Computer-Based Marketing-Simulation-Game Environment: An Empirical Investigation of the Decision-Making Process Chalk & Cheese: Executive Short-Course vs. Academic Simulations Revisiting Personality Bias in Total Enterprise Simulations Are Good Strategies Consistently Good? Investigating the Use of a Computer Simulation as an Effective Pedagogical Tool for the Application of a Strategic Model The Problem of determining an Individualized Simulation's Validity as an Assessment Tool A Simulation Based Analysis of the Value of Information in the Hrebiniak Joyce Typology of Adaptation Relative to Porter's Generic Strategies The Impact of Sales and Income Growth on Profitability and Market Measures in Actual and Simulated Industries A Comparison of a Stand Alone Version of a Simulation with the Traditional Competitive Version Computer-Assisted Gaming of International Business Analyzing Simulations with Computer-Based Programs and Applying the Experience to a Real-World Business A Preliminary Investigation of the Use of a Bankruptcy Indicator in a Simulation Environment Graduates' Views on the Use of Computer Simulation Games Versus Cases as Pedagogical Tools An Analytical Advertising Model Approach to the Determination of Market Demand Dealing with the Complexity Paradox in Business Simulation Games A Prototyping Approach for Incorporating Large Data Bases into Media Planning Simulations: An Example Using Magazine Media Through the Looking Glass, Inc: Superior-Subordinate Personality Type and the Leniency effect Evaluating the Effectiveness of Role Playing Simulation and Other Methods in Teaching Managerial Skills Student and Teacher Perceptions of a Management Simulation Course Performance Evaluation: The Effect on the Propensity to Create Budgetary Slack Management Team Formation for Large Scale Simulations Comparative Static Analysis with the Complete PPA Package: A Strategic Market Planning Tool Consistency in Intent: Learning Objectives at the 1994 Intercollegiate Business Policy Competition A New Twist on an Old Game: The Business Strategy Game: A Global Industry Simulation 3ed Evaluation of Performance in Management Simulation: A Management Coefficients Model Building SimuWorlds: Strategic Management Games of the Future Bulls and Bears: A Stock Market Simulation A Systems Thinking Paradigm and Think Computer Simulation Model of Broadcast and Cable Television Industry Competition Jacket Factory The Sales Management Simulation The Marketing Management Simulation A Demonstration of Promodel Demonstrating A New, Cross-Functional Business Simulation: Vision+ A Cost Chain For The Business Strategy Game Simulation Special Session On Experiential Teaching Compensation Dilemmas: An Exercise In Ethical Decision-Making Organizational Storytelling: Telling Tales In The Business Classroom Evaluating Experiential Training: Case Study And Recommendations The Internship Portfolio: An Innovative Tool For Experiential Learning, Critical Thinking, And Communication An Ethnographic Analysis Of The Pedagogical Impact Of Cooperative Communicating Consumer Behavior: A Long-Term Integrated Exercise Using Personal Consumption Journals And Consumer Analysis Papers An Experiential Paradigm For Teaching Business Problem Solving Developing Leadership Skills The Spss® Student Assistant: The Integration of A Statistical Analysis Program Into A Marketing Research Textbook Negotiating With Your Students Using TQM Principles To Transform Accounting Systems Into An Experiential Exercise Enhancing The Effectiveness Of Outdoor-Based Experiential Training Using Virtual Reality Concepts Case Writing In A Developing Country: An Indonesian Example Experiential Learning Using Focus Groups How Real Should Experiential Pedagogy Be? A Viewpoint From Our Students Reengineering The Internship: A New Approach To Experiential Learning Utilizing The Cosmopolitan/Local And Marginal Man Constructs To Measure Students' Propensity For Creativity Developing Experiential Processes For Teaching Quantitative Techniques For Business Team Learning Roles: A Cooperative Learning Technique Creating the Ultimate Small Business Student Experience: Melding Score/Ace with SBI The Development of Trust in Work Teams: The Impact of Touch Some Outcomes of Experiential Learning: How the Cultural Dynamics of Different Countries are reflected in Workplace Norms & Values Incorporation of Job Analysis Results in Various Forms of Selection Interviews Chudesno, Inc.: An Evaluation of an Experiential Training and Development Simulation An Exercise for Exploring the Relationship between Jungian Psychological Types and Organizational Dynamics Partnership: A Radical Approach to Experiential Learning Partnership: A Nice Idea, But How Do I Get Started? Recognizing Discrimination at Work Using Critical Incident Skills Questions to Help Students Become More Successful at Job Interviewing The Video Project Introduction to Psychological Type Theory Come On Down Understanding Facilitation for Development and Continuous Learning: A Micro-Workshop Leadership and Empowerment: An Experiential Exercise in Decision Making Experiential Exercises and Pedagogy Track Workshop: Selecting a Manager for Maquiladora, Inc. Experiential Training In Multi-Cultural Corporate Settings The Role of Facilitation as an Aid to Complete Learning An Experiential Exercise to Illustrate Difference in Information Processing Behaviors and Styles How to Deliver Accessible Survey Results Age Diversity in the Workplace- Family Feud Style Nafta Standoff: A Cross-Cultural negotiating Role-Play Three Strikes and You're Out!: A Downsizing Experiential Exercise