EVALUATION OF PERFORMANCE IN MANAGERIAL SIMULATIONS: A MANAGERIAL COEFFICIENTS MODEL Computer Simulation and Learning Theory, Volume 3, 1976 372 EVALUATION OF PERFORMANCE IN MANAGERIAL SIMULATIONS: A MANAGERIAL COEFFICIENTS MODEL Lloyd A. Swanson INTRODUCTION Management simulations as a learning vehicle have become increasingly popular in recent years, especially in schools of business or management.1 Quite often such management simulations constitute a major portion of required courses and it is not uncommon for a student to have participated in three or more such simulations prior to graduation. Usually some portion of a course’s total grade is based upon the team’s or individual’s performance in the simulation exercise. It is the purpose of this paper to report the results of research efforts conducted to assist in this grading or evaluation problem. Frequently success or failure of management simulation games in the classroom environment is dependent upon proper and fair simulation grading. Quite often the objectives given the participants are in the realm of maximizing profits in the long run. Unfortunately, the duration of most simulations is such that it is difficult to utilize such objectives as measures of performance. Consequently, considerable effort has been extended on the part of researchers to develop adequate and meaningful measures of performance. It is also felt that such measures better facilitate the educational effectiveness of simulation games. There has been some difficulty in deriving a single measure and several have indicated a preference for multiple measures of performance. For instance, Gray purports the use of a linear combination of several measures, such as market share, profit performance, inventory performance, etc., to obtain a single such measure.2 In a recent article in Management Science, Hand and Sims utilized the technique of path analysis to investigate relationships among thirteen performance criteria.3 They were able to reduce the number of performance criteria down to two, sales forecasting error and profit. At a paper presented at a recent Association for Decision Sciences meeting Butler and Parasuraman explained some of the difficulties and advantages in the use of objective multiple performance criteria for grading purposes. 4 They conclude that multiple performance criteria rankings can exert a positive influence on students by providing feedback necessary for improving several areas of decision making. however, they caution the use of mechanically used rankings of these criteria in assigning grades. 1 Widespread Use of Simulation Games,’ Computing Newsletter, Vol. VI, No. 1. (September 1972). P. 1. 2 Gray, Clifford F., Performance as a Criterion Variable in Measuring Business Game Success: An Experiment with a Multiple Objective Performance Model,” paper presented at the American Institute of Decision Sciences, Memphis Conference (April 27, 1972). 3 Hand, Herbert H. and Sims, Henry P., “Statistical Evaluation of Complex Gaining Performance.” Management Science, Vol. 21, No. 6 (Feb. 1975), pp. 708-717. 4 Computer Simulation and Learning Theory, Volume 3, 1976 373 Clearly, the utilization ~f multiple criteria has definite advantages, but many unanswered problems exist. One of the principal problems of the game administrator is not only to identify the appropriate criteria, but also to assign a proper set of weights for the different criteria in computing an overall performance grade. Additionally, there is also the problem of specifying the proper form of weights, i.e. should it be linear or of some non-linear form. Possibly there should be some relationship between the various criteria that would influence results. Then, too, there is the random element, such as sales variability, that may very well influence results and should not be ignored. The severity and nature of the competition again may play a significant role in the evaluation process. Should cardinal or ordinal values rankings be the principal measurement devices? These problems have no easy answers, nor are there any convenient guidelines available. However, an equitable and realistic performance measure for grading purposes is of considerable importance for successful educational value. A MANAGEMENT COEFFICIENTS APPROACH One possible approach to obtaining such a performance measure is to identify all the variables, determine their relationships, find a suitable measurement scale, and assign suitable weights to each. Such may be possible, but clearly a very costly effort without any guarantees of success. Another approach is to view the evaluation process more as an input-output model where the various criteria and environmental measurements act as ~inputs and the overall simulation performance grade acts as the form of the output. Thus, instead of attempting to determine and justify the form of the relationships, the weighting of various criteria, and the inter- action between the various environmental and performance measures, the game administrator’s actual past decisions are used to determine the relationships between the inputs and the output, the overall simulation performance grade. This approach is certainly more pragmatic than utopian in that it starts with the administrator’s actual decisions and builds a better system. This approach to model building or problem solving is certainly not new. In 1963 Bowman built such a model to solve the aggregate production and employment scheduling problem.5 More specifically the problem is one of determining, or making, two decisions, the size of the work force and the production rate for a month in light of forecasts for future months, inventory on hand, and the number of workers presently employed. Other approaches available at the time required a rather extensive cost collection and development of a rather elaborate mathematical model. On the other hand Bowman’s approach only employed multiple regression analysis to relate the data available to the manager at the time the decision was made to the actual decision made at that time. Paramount to developing and utilizing such an approach are two key assumptions: That the form of the cost structure is “U” or “dish” shaped, and that the manager has not been consistently “off-center” or consistently biased Thus, although, 5 Bowman, Edward H., “Consistency and Optimality in Managerial Decision Making”, Management Science, (January 1963), pp. 310-321. Computer Simulation and Learning Theory, Volume 3, 1976 374 there may be variations in decisions, the assumptions imply that experienced managers are aware of the criteria and the variables that influence the system and act accordingly. Since the results of such a mode]. are regression coefficients the approach is commonly referred to as a management coefficients model. Such an approach may very well be appropriate in certain management simulation game situations to aid in determining the grade in a classroom environment. It would only be appropriate where the game administrator was not only very familiar with the simulation and the various criteria employed, but also had gained some experience in making grades in this environment. It is also basic that the criteria be quantifiable. Thus, a management coefficients model could be developed where the various criteria or performance measures would serve as independent variables and the past grading decisions would serve as dependent variables. Then several logical relationships could be tested to determine the proper form. There are several advantages to such a method in that not only should the data be readily available, but the results can be easily computerized by simply adding a few statements to the simulation game program to print the final grade or score. Also, by employing a regression model considerable variation can be reduced in assigning grades as well as maintaining consistency not only within the class, but also from class to class and from semester to semester. In large multi-section courses such features would be most welcome. AN APPLICATION OF MANAGEMENT COEFFICIENTS FOR GRADING To demonstrate the technique and applicability of employing a management coefficients model to assist in grading student simulation performance, the approach was applied to a simulation game utilized in a required operations management course. The actual game utilized was a modified version of one developed by Vance6 a number of years ago, originally hand calculated. Besides now being computerized and the number and type of decisions expanded, the simulation has evolved an evaluation of performance quite similar to that reported by Gray.7 Evaluation points are awarded for such categories as market share, inventory management, cash management, profit performance, etc. These points as well as the sum total are reported back to the participant at the end of each decision period in an effort to identify key managerial areas and areas of concern. Although the weights, both slopes and intercepts, for each of the criteria were initially subjectly assigned, the weights were altered over a period of several years in efforts to achieve more desirable results. While there has been some satisfaction with the present weighting and no changes have been made in the last few years, there still exists some discomfort in converting the sum of the reported evaluation points into a meaningful grade. 6 Vance, Stanley. Management Decision Simulation, McGraw-Hill, 1960. 7 Gray, Clifford P., “Performance as a Criterion Variable in Measuring Business Game Success: An Experiment with a Multiple Objective Performance Model,” paper presented at the American Institute of Decision Sciences, Memphis Conference (April 27, 1972). Computer Simulation and Learning Theory, Volume 3, 1976 375 The problem was particularly difficult in this situation in that this simulation employs three company independent industries with individuals and not teams making decisions for each company. Thus there are considerable differences among the industries although the total sales available to each industry are equal. In brief, it has been observed that some industries become more competitive than others and as such two observations have been made: the more competitive industries tend to have lower average evaluation points and companies within these industries tend to do a better job in managing their companies. Thus, although the evaluation points may portray relative performance within an industry, little comparison can be made between industries. This difficulty results from the fact that evaluation points are awarded for profits and non- competitive industries have high profits, but pay little attention to other management areas. Efforts to improve the evaluation proved unsatisfactory and instead the game administrator was forced to continually make subjective decisions on grades based on a variety of factors. This consumed a rather large proportion of time and considerable frustration considering the number of students and sections. To minimize this effort and to be more consistent in grading, a management coefficients model was constructed where measures of industry competitiveness and evaluation points served as independent variables and previous grading decisions served as the dependent variable. The principal problem in this situation was in the selection or determination of the proper form of the relationship. It was felt that there were two principal determinants in arriving at grades: evolution points and a measure of industry competitiveness (derived from the average companies’ decisions within the industry). Unfortunately, it seemed that the relationship was more non-linear than linear and plotting the data seemed to bear this out. It was decided to try several non-linear forms converted to linear forms and to employ stepwise regression for variable selection. The data utilized for this analysis consisted of 83 observations from three different classes. The results from this analysis, not surprisingly, only resulted in entering but one variable yielding the following coefficient model, where G represents the numerical grade received, C the adjusted competitive index, and P the evaluation points. The correlation coefficient, r, was quite high, 0.753, and the F value, 107.06, for 82 residual degrees of freedom indicated a high significance. ANALYSIS OF RESULTS In an effort to check the validity of the structure of the evaluation points, i.e. the proper weighting of the individual criteria, the individual component was also allowed to enter the model. Since it was Computer Simulation and Learning Theory, Volume 3, 1976 376 found that none of these variables resulted in any significant reduction in variance and thus was not added to the regression equation, some credibility was given to the present evaluation. Or stated in another way, this analysis indicated that the game administrator utilized the present form of the evaluation in determining grades. To check on the validity of the model the so called predicted grades were all individually compared to the actual grades given by the administrator. Where significant discrepancies occurred a more detailed analysis was made by the administrator. The conclusion reached was that some errors were made in originally assigning grades and that the model gave better grades than the administrator. Actually this was not surprising in that a regression model yields a mean and thus reduces variability. Although the original intent was merely to assist in grading, the model has been utilized over the past year in a very dominate role in determining grades. It should be pointed out that the model is only appropriate for this particular simulation with a similar sales pattern and for approximately the same duration, 7 to 14 decision periods. It only reflects the attitude and opinions of this gate administration. Naturally if the simulation were altered, the evaluation changed, or the attitudes of the administrator shifted, then a new model would have to be developed. However, the model does offer greater consistency in grading and considerably reduced the effort of assigning grades. It was also found that although this model only reflects the decisions of this administrator, other instructors also were found resorting to the model for grade determination. Thus, such a model could be utilized to obtain grading decisions from individuals who are very knowledgeable of the simulation when the instructor utilizing the simulation is not. SUMMARY AND CONCLUSIONS In this paper a management coefficients model has developed to assist in assigning grades for management simulation games where the game administrator’s actual decisions as well as various simulation results were utilized in determining the regression model. The model developed in this paper, although only appropriate for the particular simulation employed, has shown to be very successful since its implementation a little over a year ago. It should again be emphasized that the model is more facilitating in nature than optimal as it only reflects the attitude and opinions of the game administrator. What this approach does suggest is that a similar approach can be applied to other simulations and thereby relieve one of the burdens of administrating a game, assigning grades. Such an approach should also result in having more consistent grading. It is not necessary to have an ongoing evaluation as was the case presented in this paper, rather any quantifiable simulation results that the administrator might utilize in assigning grades would suffice. Computer Simulation and Learning Theory, Volume 3, 1976 377 REFERENCES 1. Bowman, Edward IL, “Consistency and Optimality in Managerial Decision Making”, Management Science, (January 1963), pp. 310-321. 2. Butler, Richard 3. and Parasuraman, A. “Integrating Business Game Performance with the Grading Process”, paper presented at the Midwest American Institute of Decision Sciences, Indianapolis Conference, (1975). 3. Cherryholmes, S., “Some Current Research on Effectiveness of Educational Simulations: Implications for Alternative Strategies,” American Behavioral Scientist, Vol. 10 (October 1966) pp. 4-7. 4. Gerstenfeld, A. and Maynard, P., “A Study of Management Games as a Training Method.” Training and Development Journal, Vol. 25, No. 3 (March, 1971, pp. 12-15). 5. Graham, Robert G. and Gray, Clifford F., Business Games Handbook (American Management Association, 1969). 6. Gray, Clifford F., “Performance as a Criterion Variable in Measuring Business Game Success: An Experiment with a Multiple Objective Performance Model,” paper presented at the American Institute of Decision Sciences, Memphis Conference (April 27, 1972). 7. Hand, Herbert H. and Sims, Henry P., “Statistical Evaluation of Complex Gaming Performance.” Management Science, Vol. 21, No. 6 (February, 1975), pp. 708-717. 8. Klassen, C.R., “Business Gaming: A Progress Report.” Academy of Management Journal, Vol. 7, No. 3 (September, 1964), pp. 175-188. 9. Vance, Stanley, Management Decision Simulation, McGraw-Hill, 1960. 10. “Widespread Use of Simulation Games,” Computing Newsletter, Vol. VI, No. 1. (September, 1972). p. 1.