SIMULATION GAMING AS AN EXPERIMENTAL CONTEXT FOR THE STUDY OF MULTICRITERIA DECISION MAKING Developments in Business Simulation & Experiential Exercises, Volume 14, 1987 1 SIMULATION GAMING AS AN EXPERIMENTAL CONTEXT FOR THE STUDY OF MULTICRITERIA DECISION MAKING John F. Affisco, Baruch College, CUNY Michael N. Chanin, Baruch College, CUNY ABSTRACT While theoretical development of discrete multicriteria decision making (MCDM) models has proceeded apace over the last fifteen years, there has been a disappointing lack of empirical testing and validation studies. A major reason for this phenomenon is the unavailability of common decision making environments to serve as a context for such studies. This paper proposes simulation gaining as an experimental context for the study of multicriteria decision making. To illustrate the use of simulation gaming for this purpose, results of a study of two integrated MCDM-Behavioral models of group decision making conducted in tile experimental context of the Business Management Laboratory are presented INTRODUCTION Multicriteria decision making (MCDM) models follow the process-oriented approach to human decision making. According to this philosophy if one understands the decision process, one can correctly predict the outcome. Further, while essentially descriptive this approach has normative features in the sense that knowing how decisions are made can teach us how they should be made. Discrete MCDM models, also known as multi-attribute decision models, are used for selecting an alternative from among a limited finite set of options with respect to multiple criteria. Theoretical development of such models has proceeded apace over the last decade and one-half. Typical of this development has been the formulation of a class of discrete MCDM models known as spatial proximity models 11 These models relate alternatives to specified criteria through the use of spatial representations and distance measures. At the same time that major efforts have been made in the theoretical development of MCDM models, there has been a disappointing lack of empirical testing and validation studies. For example, Zeleny [13] explicitly leaves for others to attempt empirical validation of his ADAM spatial proximity model. Bao [2] reports one such effort wherein fifteen accountants were asked to chose between three capital projects with respect to three financial criteria. Results indicated that ADAM generally described choice considerably more accurately than a random process, and that the magnitude of ADAM’s ratings was a good indicator of subject’s actual judgements on capital budgeting alternatives. However, this study barely scratches the surface of the empirical validation effort that is required before implementation of ADAM may be pursued. A similar situation exists with respect to the vast majority of other discrete MCDM models. What are the reasons for the lack of validation efforts of these apparently intuitively appealing models of the decision making process? Let us postulate a few: (1) Most MCDM models have been developed by mathematical theoreticians who concentrate on modelling rather than empirical validation efforts; (2) Extreme difficulty exists in obtaining the cooperation of business or government organizations as research environments. Specifically, recruiting a sufficient number of decision makers for experimental subjects to render findings statistically significant is problematic; (3) Even if enough “real world” subjects were available, it would be highly unlikely that they would possess a common background to serve as a context for individual decision making exercises; (4) When group decision making is to be studied the above problems are compounded by the need to structure groups, observe their behavior, develop a common expertise as a decision making context, etc. The major premise of this paper is that simulation gaming provides an excellent experimental context that successfully deals with these problems of validation. We believe that this is so for the following reasons: (1) Group structures may be introduced as a natural element of the management of competing firms in the simulation; (2) Group members build a common expertise about their respective firms and industries over the course of play; (3) Simulation gaming allows for the use of and observation of the use of behavioral problem solving technologies; (4) Simulation games have been successfully utilized as a research environment for studies in various fields previously (see for example [1], [5], [6], and [9] ) (5) Instructors with experience in administering such simulations are logical candidates for conducting empirical studies due to their continuous relationship with students who engage in such decision making exercises. To illustrate this point, we will present some results from a study, conducted in the experimental context of the business Management Laboratory [10], whose aim was to empirically examine the performance of two integrated MCDM- Behavioral models of group decision making. The following sections of this paper describe the three models that were studied, the research approach, the results and some pertinent conclusions. Developments in Business Simulation & Experiential Exercises, Volume 14, 1987 2 THE MODELS ADAM-Median Model The first model under consideration is the ADAM-Median Model. This model proposed by Zeleny [14] is based on the premise that individual and group decision making are interrelated and cannot be approached from the same methodological viewpoint. Specifically, individual group members working independently rate alternatives according to the ADAM multicriteria decision making model of Zeleny [13] , and then rank them according to the smallest distance from a hypothetical ideal alternative. This involves the solicitation of a priori criterion weights from the decision maker and the calculation of an unstable context-dependent criterion weight which reflects a change in criterion importance of a particular set of feasible alternatives in a given decision situation. This last measure is defined as a traditional entropy measure. Once each individual decision maker has determined his own preference over the alternatives these ranks are utilized to arrive at a group decision. Each alternative is evaluated according to the absolute distance of its rank across decision makers from a hypothetical median ranking such that this distance is minimized. This ranking is called a median ranking by Cook and Seiford [8] and may be determined by solving a classical linear programming assignment problem. This problem is to assign alternatives to ranks so that the sum of the distances between the median rank and the actual assigned rank is minimized. ADAM-PST MODEL In the ADAM-Median model no specific consideration is given to the behavioral group processes that are part and parcel of collective decision making process. The ADAM- PST model considers group decision making processes in the development of a priori criterion weights. Specifically, groups use a Problem Solving Technology to develop group weights and these weights are used in the ADAM algorithm to derive ratings for each alternative which are then ranked according to minimum distance from the ideal. In this research three Problem Solving Technologies -Dialectical Problem Solving Technology, Devil’s Advocate Problem Solving Technology, and Nominal Group Technique - were utilized. TOPSIS-PST MODEL TOPSIS Yoon and Hwang[12] is a spatial proximity multicriteria model analogous to ADAM. There are two significant differences between them. First, whereas ADAM includes both an a priori criterion weight and a context- dependent criterion weight in its formulation, TOPSIS includes only the former. Second, while ADAM considers only the distance from the ideal in its closeness measures, the corresponding TOPSIS measure is a function of the alternatives distance from both the ideal solution and the negative (or anti) ideal solution. Once again as in the ADAM-PST model, the three Problem Solving Technologies are used to develop a priori weights. RESEARCH APPROACH This research was conducted in the context of the Business Management Laboratory [10] simulation game. This is a complex business game in which approximately thirty-five decisions are made for every quarter of play in the version used. Thirty-two groups of senior business students in the required capstone Business Policy course during the Spring 1986 semester were the subjects of this study. These groups were generally constructed of four members with different majors (i.e. Accountancy, Finance, Marketing, Management, etc.) within constraints imposed by the registration process. The groups were randomly divided into four eight team industries. Three of these industries were trained in a single problem solving technology which they were required to utilize for all decisions they made during the semester. This training consisted of: (1) lectures explaining the particular problem solving technology; (2) the viewing of a video tape illustrating the use of the problem solving technology to arrive at a quarterly decision set; (3) the use of the problem solving technology in a trial decision period followed by a critique of its use by the instructor; and (4) continual monitoring, evaluation, and feedback on the use of the problem solving technology. These problem solving technologies were operationalized according to the approach presented in Chanin [3] and Chanin and Shapiro [4] . The fourth industry was allowed to develop its own ad hoc approach to the group decision making process. No mention was made of problem solving technologies or any other group decision making methodologies. In essence, this industry served the function of a control. Each group was faced with a single decision making task. Specifically, midway through the semester, after six quarters of play, the subjects were faced with a process selection decision. This process selection decision consisted of selecting among three alternative process to replace the second stage of the group’s manufacturing facility. This decision was to be made with respect to the fifteen criteria presented in Table 1. Definitions of each criterion were distributed to all subjects and were discussed in class to ensure student understanding. in addition, their respective scales of measurement were fully explained. The decision making proceeded in two phases. In the first phase, individual decision makers were asked to rate the importance of each criterion on a 0-to-100 scale. Then at a group meeting each group was required to utilize its respective PST to arrive at a group rating for each criterion on the same scale. The second phase consisted of each individual independently rating three independent sets of three alternatives on a 0-to-100 scale, and then choosing between them by ranking the process alternatives in Developments in Business Simulation & Experiential Exercises, Volume 14, 1987 3 each set from first to third. Once again, the groups met under faculty observation and were required to rate and rank the sets of alternatives using the proper PST. These ratings and rankings are representative of group judgement and choice respectively. The instruments used to gather this data were modeled after those presented in Bao [2]. All individual and group documents were collected For all replications of the process selection decision the following were generated: (i) Individual ADAM ratings and rankings based on individual a priori weights; (ii) Group rankings based on the solution of the assignment problem to determine the median rank; (iii)Group ADAM ratings and rankings based on PST derived a priori weights; (iv) Group TOPSIS ratings and rankings based on PST derived a priori weights. These derived group ratings and rankings were then compared with the subjects’ actual decisions to determine each model’s descriptive accuracy. RESULTS Descriptive accuracy of the models was tested by measuring choice accuracy and judgement accuracy. To measure choice accuracy actual ranks assigned to alternatives by groups using a particular problem solving technology were compared with ranks generated by the three models for each decision set. Following Bao [2] a perfect match was considered accurate; any difference in ranks was considered inaccurate. For each problem solving technology the maximum possible accuracy was 24 (i.e. 3 sets X 8 groups). If ranks were randomly assigned, the probability of a correct guess was .1667. Table 2 shows that for the process selection decision the ADAM-Median model and the ADAH-PST model described group choices more accurately than a random process for all but the DAPST. Also TOPSISPST described group choices more accurately than a random process for all PSTs. Further, TOPSIS-PST generated a higher percentage of correct decisions for all of the PSTs than either of the other two models. In an effort to determine if any significant differences in the choice accuracy of the three models existed a series of Cochran’s Test for Related Observations [7] were performed. These tests compared two models at a time across all four PSTs. Results indicated that for the process selection decision TOPSIS-PST outperformed both ADAM- Median and ADAM-PST at the 0.025 level of significance. No significant difference was found between the performance of ADAM-Median and ADAM-PST. Judgement accuracy was measured by the correlation coefficient between the groups’ actual ratings and the ratings generated by the respective model. Of course, since the ADAM-Median model does not generate ratings for the group consensus, correlation coefficients should be positive since the rating is based on the relative closeness to the negative ideal solution and thus the most preferable alternative would have the highest relative closeness. All TOPSIS-PST correlation coefficients were significantly different from zero, while this was true for ADAM-PST only for the case of DPST groups DISCUSSION AND CONCLUSIONS The major thrust of this paper is to show how simulation gaming may be utilized as an experimental context for the study of group decision making models. After proposing an explanation for the lack of attempts at empirical validation of MCDM models, the paper presents the results of a study which uses the Business Management Laboratory as an experimental context to overcome these problems. The focus of this study was to compare the descriptive accuracy of three competing MCDM group decision making models. Results showed that: (1) all three models generally described choices more accurately than a random process, and (2) on a total basis TOPSIS-PST significantly outperformed the other models in choice accuracy. Further with respect to ratings, results indicated that TOPSIS-PST better described actual ratings than ADAM-PST for all PSTs. As to the question of which models better described the decision making of groups which use a given problem solving technology, no evidence was found to indicate that performance of a specific MCDM model was (or should be) linked with a specific PST. One factor that may account for this is the lack of decision maker involvement in the development of the decision criteria utilized. These results, of course, are not sufficient to draw a general conclusion as to the relative efficacy of the models under investigation. However, more importantly, this study establishes the fact that simulation gaming provides a viable experimental context for empirical study of MCDM methods. While this ,study focused on a strategic operations decision, the availability of a wide variety of function specific and total enterprise simulations presents many opportunities to study decisions that are related to the other business functions. Therefore, the utilization of free simulation as an experimental context. can spur the breadth of empirical investigation that is required before the decision sciences community can represent MCDM models as being useful to practitioners. Finally, the increasing availability of personal computer based simulations and hardware allows the development of decision making laboratories which will generate an additional impetus for the investigation of the totality of decision making processes. Developments in Business Simulation & Experiential Exercises, Volume 14, 1987 4 REFERENCES [1] Affisco, J.F. & Chanin, M.N. “A study of the effectiveness of assumption making in strategic planning.” Proceedings of Northeast American Institute for Decision Sciences, 1986, 91-93. [2] Bao, D.H. “An empirical validation of ADAM in a capital budgeting context.” Decision Sciences, 1985, 16(3), 265-268. [3] Chanin, M.N. “An empirical examination of conflict and non-conflict-oriented problem-solving technologies.” In L.A. Graff & D.M. Currie (Eds.), Developments in business simulation an Experiential exercises. Normal, Il: Association for Business Simulation and Experiential Learning, 1983. [4] Chanin, M.N. Shapiro, H.J. “Comparison of problem- solving technologies: A free simulation approach.” In D.J. Fritzsche El L.A. Graf (Eds.), Developments in business simulation and experiential exercise. Normal II: Association Business Simulation and Experiential Learning, 1982. [5] Chanin, M.N. & Shapiro, H.J. “Dialectical inquiry in strategic planning: extending the boundaries.” Academy of Management Review, 1985, 10(4), 663- 675. [6] Chanin, M.N., Wulwick, V., El Shapiro, H.J. “A study of comparative effectiveness of problem solving technologies,” In D.M. Currie El J.W. Gentry (Eds.), Developments in business simulation and experiential exercises. Normal, I1: Association for Business Simulation and Experiential Learning, 1984. [7] Conover, W.J. Practical nonparametric statistics, 2nd Ed. , New York, Wiley, 1980. [8] Cook, W.D. Seiford, L.M. “Priority Ranking and Consensus Formation.” Management Science, 1978, 24(16), 1721- 1732. [9] Courtney, J.F., DeSanctis, G., Kasper, G.M. “Continuity in MIS/DSS laboratory research: the case for a common gaming simulator.” Decision Sciences, 1983, 14(3), 419-439. [10] Jensen, R.L. El Cherrington, D.J. The business management laboratory, 3rd. Ed. Plano, TX: BPI, 1984 [11] Minch, R.P. Sanders, G.L. “Computerized information systems supporting multi- criteria decision making.” Decision Sciences, 1986, 17(3), 395-413. Developments in Business Simulation & Experiential Exercises, Volume 14, 1987 5 [12] Yoon, K. Hwang, C. “Manufacturing plant location analysis by multiattribute decision making: part I single-plant strategy.” International Journal of Production Research, 1985, 23(2), 345-359. [13] Zeleny, M. “The attribute-dynamic attitude model (ADAM),” Management Science, 1976, 23(1), 12-26. [14] Zeleny, M. Multiple criteria decision making. New York: McGraw Hill, 1982 Table of Contents Volume 14, 1987 Simulation Gaming as An Experimental Context for the Study of Multicriteria Decision Making How to Multiply Your Management Game Grading the Personnel In-Basket Exercise Simulation: The Players™ Perspective The Acute Susceptibility of Nominal Grouping to Negativity The Relative Value of the National ABSEL Meetings: An Analysis of Perceptions by Faculty and Deans A Survey of Behavioral Labs Used by American Business Schools A Structured Framework for Applying Conceptual Models to Problem Analysis: Hardening Up An Expert System Approach for Teaching Marketing Case Analysis From Theory to Practice: A Model for Teaching Beginning Advertising The Use of a Simple Forecasting Technique During an Interactive Computerized Business Game An Investigation of the Relationship Between Formal Planning and Simulation Team Performance and Satisfaction The Exponential Logarithm as an Algorithm for Business Simulations Learning macroeconomic Theory and Policy Analysis Via Microcomputer Simulation Laptop: A Principles of Marketing Simulation Teaching About Sampling in a Marketing Research Class Integrating Decision Support Systems and Business Games Demand Generation in a Service Industry Simulation: An Algorithmic Paradox ABSEL - At a Crossroads? Research on Predicting Performance in the Simulation An Expert System for Financial Planners "Developing Various Student Learning Abilities Via Writing, the Stock market Game, and Modified Marketplace Game in Beginning Macroeconomics" Professional Activity Reports: Getting back to Basics Œ A Preliminary Investigation "A Comparison of Performance, Attitudes, and behaviors of MBA and BBA Students in a Simulation Environment: A Preliminary Investigation" Decision Styles and Student Simulation Performance: A Replication Personal Power Strategies: An Experiential Exercise LTL I: A Trucking Simulation Game Total Enterprise Business Games: An Evaluation Can the Skill of Management be Taught? Developing Critical Thinking and Effective Communication Meaningful Notebooks and Enhanced Learning Using a Totally Menu-Driven Data base and Decision Support (DSS) Program Using a Computerized Menu-Driven Data Entry and fiWhat-Iffl Program as a Pedagogical Enhancement for Management Simulation Using a Joint Project Involving MBA Marketing Management and Undergraduate Marketing Research Students to Teach Marketing Research Testing the PAGE Technique: Results and Further Developments A Comparison of Student Perceptions with Accepted Expectations for Business Simulations "Conflict Management in Action: Verbal Strategies, Nonverbal behaviors and Conflict Styles" Dramatic Monologues as Surrogates for Experiential Learning Influencing One™s Superior: An Assessment and Exercise Personality Traits and Organizational Cultures: Two One-Hour Experiential Learning Exercises Computype: A Strategic Marketing Game Open System Simulations and Simulation Based Research "Teaching Forecasting to the Masses: Little Decision Support Training, But a lot of Answers" Goal Setting and Performance Evaluation with Different Starting Positions Œ The Modeling Dilemma The Gamesmanship of Pricing: Building Pricing Strategy Skills Through Spreadsheet Modeling Teaching MPR Experientially Through the Use of Lotus 1-2-3 Business Simulation Through Activities of a Manufacturing Company Simulated Stock Broker Teaching Employee Counseling Skills to Management Students: A Simulation Airline: A Strategic Management Simulation Developing Simulations and Experiential Exercises on the Personal Computer: Some Critical Issues A Powerful Tool Œ For What? An Integrative Systems Approach to Simulating Free Enterprise in a Small Business Course in a Business School Profits: The False Prophet The Use of Expert Systems to Develop Strategic Scenarios: An Experiment Using a Simulated Market Environment An Interdisciplinary Workshop: Spreadsheet Modeling and Case Analysis Techniques for Beginning Executive MBA Students An Innovative Approach to an O. D. Class Œ The Creative Interaction The Application of Spreadsheet Software Technology to Complete Taxpayer Elections Experiential Learning as the Cornerstone of an Industry Management Course "Identifying, Exercising and Enhancing Right Brain Skills in the Business Classroom" Utilizing Envisionary Technology in the Business Policy Course: An Experiential Pedagogy Business Simulation in the Policy Course: A Survey of American Assembly of Collegiate Schools of Business The Moving Van: Vehicle to Success of Disaster for the Two-Gender Work Force Oil and Gas Well Investment Analysis Using the Lotus 1-2-3 Decision Support System Team Cohesion Effects on Business Game Performance