OPERATIONALIZING A TEST OF A MODEL OF THE USE OF SIMULATION GAMES AND EXPERIENTIAL EXERCISES Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 48 OPERATIONALIZING A TEST OF A MODEL OF THE USE OF SIMULATION GAMES AND EXPERIENTIAL EXERCISES James V. Gentry. Oklahoma State University Alvin C. Burns, Louisiana State University ABSTRACT The authors propose an addition of two important dependent variables: Student Learning and Student Attitudes, to a previously-reported model of the use of simulation games and experiential exercises. Additionally, a research design proposal is tendered for ABSEL members’ inspection and criticisms. The authors specify the anticipated operationalizations of variables, tests for reliability and validity, and the general analysis of the model’s postulated relationships. The paper is couched as a necessary step intermediate to theorizing and empirical hypothesis tests. INTRODUCTION Burns and Gentry (1977, 1980) have attempted to specify a number of variables assumed critical to the success of simulation games and experiential exercises and place these variables in a causal framework, Their conclusion, arrived at through causal modeling, was that “user Involvement” is the most crucial variable of the twenty-five specified, as it is hypothesized to relate, either directly or indirectly, to all others. However, the model proposed by Burns and Gentry (1980a) considers only independent variables and fails to explicitly relate the independent variables to salient dependent variables: namely, student learning and student attitudes. Recently, Burns and Gentry (1980b) have restructured the hypothesized linkages between the independent variables and postulated impacts on these the dependent variables. Given that the model has been hypothesized, the logical next step is to test the hypothesized relationships empirically. This paper will focus on a necessary intermediate step between theorizing and hypothesis testing that of developing the research design. Admittedly, it is tempting to go ahead and acquire data from users and participants; however, the intent of this paper is to propose a comprehensive research design that would preclude a “quick and dirty” data collection effort. Given the care taken in the development of this topic thus far, it would be inconsistent to ignore the tenets of proper research. Moreover, an earlier attempt (Gentry, McCain, and Burns, 1979) to operationalize the Bloom taxonomy (Bloom et. al., 1956) showed vividly that the operationalization stage is critical in educational research, for the results indicated a high level of confusion as to the nature of the underlying concepts. Consequently, rather than to rush the data collection process and obtain data of questionable value, our intent is to use this paper to introduce our proposed methodology with the objective of electing constructive feedback from ABSEL members. DEPENDENT VARIABLES IN TME MODEL Burns arid Gentry (1977) described the independent variables hypothesized to be related to the effectiveness of simulation games and experiential exercises. These brief descriptions are provided in Figure 1. The two dependent variables, Learning end Attitude, are discussed below. Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 49 Learning - One must assume that the educational system exists to develop the student. As such, learning is critical variable of concern when measuring the effectiveness of a particular teaching methodology The operationalization of “learning” poses difficult measurement problems, however, many different measures have been used to represent “learning,” including: simulation game or exercise performance, objective test performance, course grade, and self-reports. In fact, the definition and measurement of learning proves to be an elusive goal. Confounding occurs in most instances. For example, simulation or exercise performance measures both knowledge of the concepts and am understanding of the game itself. The use of standardized tests dealing with the general course topic removes this problem, but introduces others. One such problem occurs when the teaching methodology is used with one objective in mind, and the test measures performance not directly related to it. Bloom et al. (1956) developed a classification system of six different levels of learning. The Bloom hierarchy structures learning in progressive strata: Basic Knowledge, Comprehension, Application, Analysis, Objective Synthesis, and Objective Evaluation. Figure 2 summarizes each level. It is not uncommon to find an objective test emphasizing Basic Knowledge being used to measure learning in a course in which simulation game was employed. As the objective of simulation game would be Application or possibly Analysis, it follows that Basic Knowledge measures are inappropriate. Clearly, the level of learning sought should be determined very early. If, in fact, the purpose of the course is to provide an awareness of the general topic area, then methodologies aimed at higher levels of learning may be counter-productive. On the other, if the objective is to improve the student’s ability to apply the concepts, the use of multiple choice questions from the typical instructor’s manual is also inappropriate. Consequently, one must be extremely careful in how he/she defines the dependent variable “learning.” in short, the chosen pedagogy must be attuned to learning objectives and results measured by proper tests. Attitude - “Attitude” refers primarily to the student’s affective reaction to the teaching methodology, although the term is often used to encompass the affective reactions to the instructor or to the course. Student attitudes have also received a great deal of attention for several reasons. One reason is that student attitudes are often postulated as an intervening variable between the pedagogy and learning; highly motivated students are likely to learn more. A second reason for the emphasis on attitudes is their use as the measure of teaching ability (in the form of teacher evaluations) in administrative reviews of faculty. Student attitudes, then, represent a tangle of reactions to pedagogy, teacher attributes, and course context. Seemingly inexorable interactions undoubtedly exist. A relevant consideration is the temporal aspect of the measurement. Most measures are taken either after the termination of a particular exercise or at the termination of the course. Lt has been argued that the proper time to measure the perceived effectiveness of teaching methodology is after the student has been in the business world for a few years. While the authors agree that perceptions are more valid after some period of time has elapsed, we will concentrate on a measure of attitude taken in the short run due to: (1) the greater feasibility of measurement and (2) the necessity of short-term feedback in most pedagogy decisions. PROPOSED RESEARCH DESIGN The study will consist generally of surveying (1) faculty and (2) students, using the questionnaire shown in general for in Appendix A (Faculty version). The questionnaire represents our proposed operationalization of the Several variables under study. Faculty Population. Many of the independent variables arid, in fact, the 1980 version of the model (Burns and Gentry, 1980) have an instructor-orientation; that is, the effectiveness of the particular teaching methodology is evaluated by the instructor. Consequently, a survey of faculty using simulation games and experiential exercises will be made. While the model of the use of the methodologies is proposed as a general one, we hypothesize that the strength of the relationships will vary according to the following variables: Methodology Used: Simulation Games vs. Experiential Exercise Duration of the Exercise: Short vs. Long Scope of the Exercise: Operational vs. Strategic Nature of the Exercise: Stochastic vs. Deterministic Student Population: While faculty have their own perceptions of the effectiveness of their teaching approaches, students must be studied inasmuch as the dependent variables consider their learning and their attitudes. Moreover, it is highly likely that there will be discrepancies between faculty and student perceptions of the various variables (even the ones that can be measured objectively, such as the number of decision variables, the duration, the number of students, the resource base, etc.) due to the different frames of reference. Also, nature of the relationships found among the variables are hypothesized to vary with the following student-related factors (as well as those specified for the faculty population): Background: Business vs. Non-Business Major Level: Freshman-Sophomore vs. Junior vs. Senior vs. Graduate Student Sex: Male vs. Female Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 50 Time Status: Part-time vs. Full-time Sample Design. We propose to use the ABSEL membership as a first source of faculty responses (and student responses, as we will ask certain ABSEL members to survey their students). Unless sufficient responses are obtained for all of the various cells, we will sample using adoption lists for simulation games and experiential exercises from cells needing more response. Proposed Analysis. Proper instrument development procedures dictate that the measuring device be assessed for reliability and validity. A customary approach to these concerns is the multitrait-multimethod matrix method advocated by Campbell and Fiske (1959) . Essentially. this approach requires that reliability be ascertained through test- retest procedures. Validity is apparent when high, positive correlations are deter- mined for the same variable as measured by maximally different methods. The issue at hand, then, becomes the determination of methods which are sufficiently dissimilar yet comparable. Were it not for the previously stated belief in the differences in the perceptions of faculty users and students, it would have been acceptable to consider the two populations as dissimilar methods. However, the contention requires that reliability and validity be determined within populations. We propose to draw subsamples in addition to the primary sample from each population and to administer instruments with different wording and different response scales. Thus, convergent validity will be assessed with test- retest measures. Upon satisfactory results from the reliability and validity measurement phase (We realize that this phase may require a number of iterations.), attention will turn to tests of the various postulates and hypotheses within the general model. The model, one will recall, is a set of expected causal interactions among the several variables. Unfortunately, the huge aggregation of variables and relationships renders experimentation impractical; consequently, it is necessary to look to some other device for aid. It would seem that the most likely candidate in this regard is path analysis, which has been used unsuccessfully to isolate causal relationships in associative data. CONCLUDING COMMENTS The reader may experience frustration with our description of the proposed analysis; however, it is the intent of this paper to focus attention on the operationalizations of variables proposed research design. In this interest, we earnestly solicit the aid of ABSEL members and interested others in the critical evaluation of our instrument and basic procedure. It is our firm intention to incorporate constructive criticisms and useful suggestions in questionnaire revision before field tests commence. At the same time. any observations or shared experiences by ASSEL members with regard to the expected differences in perceptions within and between the two populations under study will be appreciated. We anticipate a comprehensive progress report for ABSEL next year as the next step in our long- term, programmatic effort to isolate the salient variables and relationships in the successful use of simulation games and experiential exercises. REFERENCES [1] Bloom, B. S., N. D. Englehart, E. J. Furst, V. H. Hill, and D. R. Drathwohl (1959), Taxonomy of Educational Objectives: The Classification of Educational Coals. Handbook 1: Cognitive Domain. New York: David McKay Company. Inc. [2] Burns, Alvin C. and James V. Gentry (1977), “Some Thoughts on a ‘Theory’ of the Use of Games and Experiential Exercises,” Proceedings, ABSEL, Dallas, 17-20. [3] Burns. Alvin C. and James W. Gentry (1980a), “Moving Inward a ‘Theory’ of the Use of Simulation Games and Experiential Exercises,” Proceedings, ABSEL, Dallas, 17 [4] Campbell, Donald T. and Donald W. Fiske (1959), “Convergent and Discriminate Validation by the Multitrait-Multimethod Matrix,” Psychological Bulletin, March, 1959, Vol. 50, No 2, pp 81-106. [5] Gentry, James V. and Burns. Alvin C. (1980b) “Restructuring A ‘Theory’ of the Use of Simulation Comes and Experiential Learning,” Working Paper, College of Business Administration, Oklahoma State University. [6] Gentry, James W. Kenneth C. McCain. and Alvin C. Burns (1979), “Relating Teaching Methods with Educational Objectives in the Business Curriculum,” Proceedings, ABSEL, New Orleans, 196-198. Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 51 Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 52 Table of Contents Volume 8, 1981 The Promotion: Human Sexuality in Organization The Simulation of Chaos Leadership Development in a Simulated Urban/Suburban (U/S) Environment Tomed: A Computer Game Emphasizing Social Responsibility/or/why the Pop-Top Can? Integrated Brain Activity and the Manager's Job: Utilizing the Troika Model What does R2 Have to do with a Product Management Course? An Analysis of the Effects of Jungian Problem-Solving Style Dimensions on Marketing Decisions Bargaining Behavior in Personal Selling and Buying Exchanges Extending the Simulation Product Life Cycle A Generalized Algorithm for Designing and Developing Business Simulations Operationalizing a Test of a Model of the Use of Simulation Games and Experiential Exercises Pygmalion and Perception: An Experiential Exercise Behavioral Consequences of Reward Regarding Employee Absenteeism in an Industrial Setting: An Operant Conditioning Approach The Simlab Program: The Use of Experimental Simulation and Process Analysis for the Development of Management and Organizations Designs for Research on Simulation-Games, Cases, and Other Experiential Exercises The Role of Students in The Case Method Weaknesses in Research Design Critical Variables in Research on the Educational Value of Management Games Research Questions for Cases Research on the Learning Effectiveness of Business Simulation Games - A Review of the State of the Science The Effects of Valuation Techniques on Holding Cost During Inflationary Periods: A Simulation Exercise The Operations Simulation - A Study in Game Development Applying Guided Design to the Production/Operations Management Course: A Progress Report and Evaluation Terminal Data Entry and Retrieval Systems Simulations and Microprocessors Microcomputers and Related Technology for Simulation Gaming Microcomputers - A New Technology for Innovations in Business Simulations Microprocessor Controlled Interactive Video Simulation Business Game Design: From Theory to Practice The Success of a Computerized Simulation in Microeconomic Pedagogy The Test Preview Game: Applying the Game Show Format Providing a Real World View of the Personal Function: A Simulation Finding an Effective Means of Teaching Managerial Behavioral Skills: Two Different Experiential Teaching Methods Compared A Management Development Program Based on the Experiential Learning Model Participant Type Differences in Response to Experiential Methods: An Informal Look Preparing Student Groups to Participate in Experiential Group Projects: An Organizational Development Approach An Instrument for the Assessment of Learning Dimensions: A Progress Report of the Learning Dimension Scale (LDS) An Empirical Analysis Relating the Learning Style Inventory to Memory and Logical Ability Teaching Styles in Simulation Experiential Learning Versus Traditional Teaching Styles Student Perceptions of Effective Teaching Behaviors Problems in Evaluation of Experiential Learning in Management Education Students' Perceptions of Learning by Simulation A Relative Evaluation of Experiential and Simulation Learning in Terms of Perceptions of Effected Changes in Students Overview of Computer Based Business Games in Business Policy Classes Behavioral Decision Theory and Business Policy Giving Accounting Students Writing Experience as Job Preparation Getting to First Base with MBO: An exercise for Writing and Evaluating Objectives Dimensions of Conflict in Experiential Learning Consumer Alienation and Perceived Relevance of the Business Simulation Using the Self-Reference Criterion to Simulate Culture in Internationalized Business Course Experiential Learning in a Cross Cultural Setting- The Practice of Simulation Approach to Business Education in Japanese Universities Student' Perceptions of the Use of a Computerized Simulation in Teaching Management Information Systems Using the Case Study Approach to Develop a Microcomputer Bases, Fully Integrated, Data Base Driven, Management Information System The Case Study as a Tool for Organizational Change: Applying the Steel Ax to the Designers of Management Information Systems Toward a Theory of Teaching Business Policy A Case Study in the Use of Experiential Learning (A Management Game Simulation) to Enhance Student Understanding of Strategy Evaluation and Policy Formulation Suggestions for Integration of the Business Administration Core Publishing Opportunities and Requirements for Business Simulation and Experiential Learning Materials How do we Apply Experiential Learning Intercollegiate Case Competitions for M.B.A. Students: Initiation and Implementation Teaching Business Policy and Strategy Using the Incident Process The Learning Co-Op Approach to the Core Policy Course Meeting the Managerial Skill Shortage - Is Academia Up to It? An Empirical Analysis of Experiential Learning Reinforcement International Experiential Learning: Experience is the Best Teacher Encouraging Student Participation During International Academic Programs European Summer Study Program: Can you, Should you, What Does it Take? Travel Seminars in Europe: How can I Direct One? International Experiential Learning: Student Evaluation ABSEL: Empirical Findings on the State of the Association Sensitivity of Performance Scores in Business Simulations Improving the Learning of a Business Simulation Game by Increasing the Process Content Student Participation in Deciding Performance Criteria for Grading in an OB Course: An Exercise and a Case Study Markup for Profit: A Simulated Self-Administered Experience in Retail Pricing The Investment Decision Game: An Experiential Learning Approach to Stock Market Decisions Through Gaming CHIPS: A Marketing Channels management Game External Validation: An Experimental Approach to Determining the Worth of Simulation Games Teaching Performance Appraisal Skills: An Experiential Approach In Support of Experiential learning: Results of a Follow-Up Survey The Introductory Management Course: Taking Theory Application One Step Further Decision Efficiency and Effectiveness in a Business Simulation The Implications of Cognitive Processing Variables and the Complex Decision Simulating the Simulation for Enhanced Player Rationality Simulation/Experiential Learning Audit