A METHODOLOGY FOR MEASURING DECISION-MAKING IN A BUSINESS GAME Simulation Games and Experiential Learning in Action, Volume 2, 1975 199 A METHODOLOGY FOR MEASURING DECISION-MAKING IN A BUSINESS GAME Roger Calantone, University of Massachusetts and Gary Grudnitski, University of Massachusetts INTRODUCTION In most learning situations, an instructor can choose from a variety of pedagogical methods. These methods might include: lecturing, case studies, programmed learning, management gaming, or some combination of these methods. The primary advantage of lecturing, case studies, and programmed learning is the feedback provided for evaluating performance and guiding remedial work. That is, the feedback spawned by these methods is both descriptive and prescriptive. It has been argued, on the other hand, that this advantage is largely offset by the sterile and artificial context prevailing when the point of emphasis is decision-making. Problems or situations associated with these methods are usually cast in a static world, wherein decision parameters enhance the property of explicit definition or logical indication. The “live” case is perhaps the ideal method to teach aspiring managers effective decision- making, since the learning experience, in a decision-making/feedback context, is most realistic to the individual. Such experimentation, however, can be prohibitively costly. Proponents of management games have argued that such games offer a viable alternative, negating, to a large extent, many shortcomings of other learning environments. Management games, from the standpoint of enabling the participant to practice making decisions, are dynamic, since to some degree, they embody the complexity of a real-world environment. While we concur with this argument, we also wish to point out that the adage of “practice makes perfect,” does not always hold. In fact, the mere practice of decision-making is not a sufficient condition for development of better approaches to, or outcomes from, the decision-making process. One cannot assume that out comes are being incorporated, by the individual, as feedback in modifying and restructuring his decision process model. While some may maintain that this critical weakness stems from a functional fixation with results, rather than the reasons for these results, we contend that, more likely, it has been due to our inability to accurately assess or measure an individual’s decision-making structure. The purpose of this paper is to outline a methodology to assess the conceptual framework of a Simulation Games and Experiential Learning in Action, Volume 2, 1975 200 decision-maker, and illustrate its application in a managerial gaming environment. Further more, we will suggest a framework for validation of these derived conceptual models. METHODOLOGY The methodology used in this paper is an adaptation of the Role Repertory Test (Rep Test), developed by George Kelly [5] to obtain, within a clinical environment, role constructs in subjects whose personal-social behavior was of concern. The Rep Test has received widespread attention in Great Britain [2] [3] [6] [7] in the area of consumer research. In the area of decision- making, the most extensive experimentation using the Rep Test has been done by Jarrod Wilcox [8]. Specifically, Wilcox investigated Kelly’s methodology with respect to measuring assumptions held by professional investors as to the importance and relationship of attributes of common stock. We used an adaptation of the Rep Test to detect the types and sources of information used by decision-makers, as well as to assess the reasons for the importance of this information. The particular business game we selected for application of this methodology, emphasized planning and control activities based on accounting data. The game players were drawn from three sections of the Undergraduate Managerial Accounting course offered at the University of Massachusetts in the Fall Semester of 1974. Of the 65 participants, 17% were seniors, 67% juniors, and 16% were sophomores. From these students, 19 teams were randomly formed. Specifically, there were five steps followed in the data collection procedure: 1. Following the third decision period, a questionnaire was distributed to all participants. The intent of the questionnaire was to elicit the role played by various types or sources of information in the decision-makers’ conceptual structure of the game environment. In total, sixteen questions were asked. Examples of the questions were: “A source of information you consider crucial to the planning process ,“ “A source of information you consider misleading,” and so on. Six of the questions related to planning decisions, two to control decisions, and the remaining eight, to general considerations. 2. From the questionnaires, a limited number of triads of information sources were formed. The intent of triad formation was to elicit, for differing combinations of information sources, a pattern of relationship that was Simulation Games and Experiential Learning in Action, Volume 2, 1975 201 either similarity (construct) or dissimilarity (contrast) based. Each information source appeared in exactly three triads. Although the choice of three was somewhat arbitrary, it was influenced by the tradeoff of participant fatigue versus adequate comparison exposure of each of the information sources. Triad order of presentation was done on a random basis. Similarly, both the ordering of information sources within the triads, and triad formation itself were randomized, with the sole consideration that no pair of information sources appear in more than one triad. 3. Each group was then interviewed. The dialogue was tape recorded. In most instances, the interviews lasted from 25-40 minutes. 4. Upon completion of all interviews, the tapes were reviewed. Polar adjective scales were formed for each construct. 5. The initial questionnaires and sets of adjective scales were then returned to the individuals with the request that each information source be ranked on each adjective scale. For those information sources inappropriate to a specific adjective scale, a category labeled “Scale does not apply” was provided. To remove redundant adjective scales, each team’s set of scales was factor analyzed. Because the data was less than internally scaled, a nonmetric, rank reducing algorithm, (SSA-III, [41) was used. SAMPLE RESULTS Contrasted are the results from two competing teams, (see the Appendix for details). On an ordinal basis, team A placed at the top in terms of profit and planning performance while team B finished near the bottom on both of the evaluative measures. We attribute much of this difference in achievement to the striking dissimilarity of information use on the part of the teams. Team A relied heavily on tools which facilitate effective planning and control. Furthermore, many of the information sources were perceived to be of an interrelated, macro nature. Team B, on the other hand, relied on subjective estimation of the more important variables, allocating an inordinate amount of importance to aspects that had less than a dramatic impact on the final outcome. Simulation Games and Experiential Learning in Action, Volume 2, 1975 202 Team A 1. Team A placed heavy emphasis on budgeting and product price determination. The attributes attached to these information sources, as derived from the factor analytic solution, were: interrelatedness, effectiveness in evaluating alter- natives, ability to track and compare data over time, related to sales and finally, items which were profitable to experiment with. 2. Team A correctly deduced inventory and carrying costs a~ items best worked “backward from;” research and development and industry sales were items worked “forward to.” 3. Projected industry demand and sales trends were sources of information highly susceptible to misinterpretation, and therefore, should be approached with due caution. 4. Finally, team A perceived the existence of a dicototny between planning and control activities. As such, both the comprehensive budget and industry sales estimation were viewed as information sources related directly to the planning process; inventory costs and overall production costs were seen as deserving of attention only in a control sense. Team B 1. Team B relied heavily on team interaction for the determination of important variable levels. Group consensus or opinion was the prime vehicle used to arrive at decision variables such as product price and promotion. 2. Team B devoted their remaining decision effort to control aspects. Idle capacity and cash position were two such items of concentration. EXPLORATIONS IN METHODOLOGY VALIDATION Once the decision-maker’s information sources and adjective scales that appeared to make these information sources relevant were elicited, the question that must then be asked is, how well have we captured the decision-maker’s conceptual structure? One way to answer this question would he to pre sent the decision-maker with his derived conceptual structure and request a response as to its appropriateness. Validation using this approach is tenuous, as an individual may be unwilling to admit to a highly simplistic structure, he confused by a very complex structure, or be unaware on a conscious level of many of its aspects. Simulation Games and Experiential Learning in Action, Volume 2, 1975 203 A more fruitful approach would be to compare the decisions expected from decision-makers having a particular conceptual structure against those decisions actually made. If there is no significant difference, we could assume our methodology captures many important aspects of a decision-maker’s conceptual structure. In practice, this approach can follow two paths. Wilcox [8] used the initial data to build a structure which was used to make predictions on the manner of handling new stimuli. The implicit assumption here is that both data sets are homogeneous, and that the model is relatively insensitive, over time, to learning or environmental effects. The second path concentrates on a single data set, thereby freeing itself from the above assumption. What is necessary however, is an a priori, hypothesized structure for evaluation. Clearly the strength of this path is directly related to the validity and generality of the hypothesized structure being the true state of nature. In the area of planning and control systems (especially germane to the gaze we used) ,Robert Anthony [1] has proposed a frame work which consists of three elements: strategic planning, management control, and operational control. Although these elements or subsystems are clearly related, because each has a different purpose and set of characteristics, a distinctive way of thinking about each is required. Specifically, Anthony proposes distinguishing the planning and control elements on the basis of the complexity and the nature of information relied on. He feels that strategic planning involves the consideration of many variables and relies heavily on external information sources, or internally generated information that has been recast to fit the needs of the problem being analyzed. This can be contrasted to control decisions which entail far fewer variables and thus, a less complex process. Given the derived conceptual maps, Anthony’s structure allows us to formulate the following hypothesis: Those decision-makers having more complex cognitive maps, and relying more heavily on externally generated data or internally transformed data, will out perform in a planning sense, those decision-makers having less complex cognitive maps, and relying less on externally generated or internally transformed data. Presently, we are attempting to quantify our measures regarding planning performance and cognitive complexity so that this hypothesis can be tested. Simulation Games and Experiential Learning in Action, Volume 2, 1975 204 SUMMARY A methodology for measuring the conceptual framework of the decision-maker in a managerial gaming context has been explained and its use illustrated. In addition, a possible framework for validation has been outlined, and a validation hypothesis generated. Simulation Games and Experiential Learning in Action, Volume 2, 1975 205 REFERENCES [1] Anthony, Robert N., Planning and Control Systems: A Framework for Analysis (Boston: Division of Research, Harvard Business School, 1965). [2] Clemens, John and Thorton Crossley, "Evaluating Nonexistent Products,” Admap, Vol. 4 (April 1968), p. 5. [3] Frost, W.A.K., “The Development of a Technique for T.V. Program Assessment,” Journal of the Market Research Society, Vol. 11 (January 1969), pp. 25-44. [4] Guttman, Louis and James Lingoes, “Nonmetric Factor Analysis: A Rank Reducing Alternative to Linear Factor Analysis,” Multivariate Behavioral Research, (October 1967), pp. 485-505. [5] Kelly, George A., The Psychology of Personal Constructs, Vol. I, A Theory of Personality (New York: Norton Press, 1955). [6] Lunn, J. A., “Perspectives in Attitude Research: Methods and Applications,” Journal of the Market Research Society, Vol. 11 (July 1969), pp. 201-213. [7] Sampson, Peter, “Using the Repertory Grid Test,” Journal of Marketing Research, Vol. 9 (February 1972), pp. 78-81. [8] Wilcox, Jarrod W., A Method for Measuring Decision Assumptions, (Cambridge, Mass.: MIT Press, 1972). Table of Contents Volume 2, 1975 ABSEL Research - From Adolescence to Adulthood, Framing the Future of Business Simulation and Experiential Learning Inventory Simulation - A Time Sharing Television Output Simulation OMSIM: An Operations Management Game An Experiential-Cognitive Methodology in the First Course in Management: Some Preliminary Results Experiential Training Methodology, Traditional Training Methodology, and Perceived Opportunity to Satisfy Human Needs Operational Problems and Solutions of Business Gaming: A Primer One Experience with the V. K. Gadget Company - An Introduction to Managerial Accounting A conversational Marketing Mix Exercise The Use of Program BAYES in the Teaching of Sample Size Determination in Survey Research An Experiential Study of Performance in a Basic Management Course Using Student Opinions in Evaluation Results with a Business Game Some Impacts of Varying Amounts of Information on Frustration and Attitudes in a Finance Games Player Performance under Differing Player Configurations in the Investment Game: Some Preliminary Observations Integrating Across Functional Areas with a Computer-Assisted Case Using Computer Assisted Cases for Marketing Research Instruction ACQUIRES: ACcounting QUick Information REtrieval System A Simulation Approach to Data Processing Controls Motivating Simulation Game Performance and Satisfaction with Group Performance-Contingent Consequences The Educational Impact of Supplementary Personal Interaction in Computerized Business Games The Minnesota Manpower Management Simulation Games RAISE II, A Personal Simulation Management in a Test Tube: A Small Group Laboratory Simulation A Methodology for Measuring Decision Making in a Business Game A Case Study of a Capital Investment Simulation Combining Experiential and Clinical Methodologies in a Small Business Management Program Results of Using Gaming to Teach Ethics and Social Responsibility Business and Society: An In-Basket Simulation Crisis versus Non-Crisis Simulation Gaming Warm-up Company: A Business Simulation Business Simulations: Competition or Learning A Computerized Management Training System for Franchized Dealers Mode I Stores, Inc.: Computer Supported Cases on the Marketing Research and Problem Solving Process Experiential Learning in Statistics Through the Computer The COMPUSTAT Analysis System as an Instructional Resource Research on Experiential Learning: Enhancing the Process A Comparison of Lecture-Case Study and Lecture-Computer Simulation Teaching Methodologies in Teaching Minority Students Basic Marketing Student Evaluation of Reaction to a Marketing Simulation Game Under Varying Circumstances The Validity and Usefulness of Packaged Models in Game Play The Effectiveness of Experiential Methods in Training and Education: A Review Guidelines for the Future Development of Business Games The Future Potential of Structures Learning Exercises Turning Them on to Management by Turning Them Out to Managers with Video Tape Recorders