A METHODOLOGY FOR ASSESSING THE INTERNAL VALIDITY OF BUSINESS SIMULATIONS Developments in Business Simulation & Experiential Exercises, Volume 10, 1983 A METHODOLOGY FOR ASSESSING THE INTERNAL VALIDITY OF BUSINESS SIMULATIONS Robert E. Schellenberger, East Carolina University John C. Keyt, East Carolina University INTRODUCTION AND PURPOSE Business simulations and games are popular teaching methods in management education today. The formation of ASSEL and the volume of dollars spent (estimated to be $100,000,000 a year in 1975 by Neuhauser [11]) attest to this fact. However, the question of value i.e. “What do simulations and games do for the student?” continually make the user. The purpose of this piper is to provide a methodological fr4mework for more rigorously assessing the value of business simulations. To accomplish this objective, the paper will: I) discuss and use a training perspective as a starting point in design of assessment methods, 2) discuss the use of experimental research tools and their use in assessment, and 3) present a brief proposed application of the assessment framework. THE TRAINING PERSPECTIVE Since management education is for all practical purposes a training situation, the use of i simple conceptual representation of training provides a useful point of view for more rigorously designing and assessing business simulations. This model was developed specifically from .3. P. Campbell [21 and R. M. Gagne [4] and generally from Goldstein [61. Figure 1 portrays this simple training paradigm as applied to management education. The paradigm shows th (the college of universit program composed of t Those skills obtained f transferred to the work to attain some level o major objective of the job performance by pro and attitudes that can be Designing a training s several steps. The first s the forces (internal an produce good job perfo broken down into tho reasonable control over uncontrollable. Those forces in the controllable group then become the focus of the development of the training program,. The second step is then to develop the training program. Based on the assessment of the job success factors, training gods ire developed. From these training goals, methods are selected and specific activities are developed which are expected to achieve the training goals. The final step in the training process is an assessment of the training program. Assessment can take place at two points in the training paradigm. Assessment can take place at the completion of the training program (still in the training environment) and on the job (the work environment). Both assessments are necessary for d successful training development program but for different reasons. The assessment completed in the training environment seeks to determine if the training methods selected And the specific activities developed, were successful in achieving the training goals. The assessment in the work environment seeks to determine the appropriateness of the training goals (and ultimately the analysis of the job performance factors- step one) and the transferability of the skills and attitudes of the training program to the actual job. Applying this training development process to business simulations first requires an assessment of the factors which cause good job performance by managers. The conceptual model of Manager Effectiveness presented in Campbell et al [3] is probably the most comprehensive identification of factors which are related to and presumed to cause good job performance. Un- fortunately, its conceptual and comprehensive characteristic makes direct use of it difficult. However, it does provide a framework leading to TRAINING AS APPLIED AS A L Training Program (Business Simulation) TRAINING ENVIRON FIGURE 1 TO BUSINESS SIMULIATIONS EARNING TOOL at within the training environment y) the trainee experiences a training raining goals and training methods. rom the training program are then environment and used by the trainee f successful job performance. The training paradigm is to insure good viding appropriate and usable skills transferred to the work place. ystem to achieve this god requires tep must include in understanding of d external to the individual which rmance. These forces must then be se which the individual his some and those that are for the most part factor groupings to be recognized in understanding managerial effectiveness. Shown below as Figure 2 is Campbell's model. The major elements include: i person (the manager) utilizing a process (job behavior) to attain a product (organizational results). Thus certain characteristics possessed by the successful manager coupled with certain behavioral factors lead to successful organizational results. Trainee’s Job Performance WORK ENVIRONMENT Trainee (Student) MENT SCHEMATIC PORTRAY MANAGE ORGANIZA Individual Characteristics Intelligence Aptitude Knowledge Temperament Preferences Expectations (ability, m N 6 FIGURE 2 AL OF THE TETERMINERS OF RIAL BEHAVIOR TION ENVIRONMENT Organizational Results Profit Maximization Organizational efficiency High productivity Job behavior otivation, opportunity) Feedback T PROCESS Feedback PERSO PRODUC Developments in Business Simulation & Experiential Exercises, Volume 10, 1983 Our interpretation of Campbell coupled with a recognition of the need to measure the factors leads us to the following simplification of the model. Factor categories to predict successful managers are: 1) job (technical) knowledge, 2) basic intelligence, 3) problem solving ability, 4) values and 5) traits. We are also indebted to the cited works of Mahoney [9, 10] and Ghiselli [5] in proposing these categories. Clearly these variables represent varying degrees of controllability by the individual manager. Basic intelligence and traits seem to be a given for an individual manager. Job (technical) knowledge and problem solving abilities, and to a lesser degree values, would seem to be more controllable and thus teachable variables. These three categories should then become the basis for determining the educational goals of business simulations. Schellenberger [14] calls the educational goals learning variables and notes three general categories. First , attitudes such as an administrative point of view or tolerance for ambiguity that are gained by game play The second category noted is knowledge. Knowledge refers to knowledge learned or enhanced by game play, such as sues forecasting (technical knowledge). The find learning variable category refers to skills. These are defined is integration and problem solving. The next step in this application would be the selection of training methods and specific tasks. The assumed choice for method is, of course, a business simulation However, the tasks required by business simulations can vary widely, not only in number, but in complexity. Thus, the game builder must balance the game tasks against the training goals in construction of the business simulation (see Schellenberger [14] for a discussion of these tasks). The find step is the assessment of the training value of the game. As specified earlier, assessment takes place at two points in the training paradigm. The first point is after the completion of the simulation experience. This point of assessment is often termed “internal validity” in the literature and this paper will define “internal validity” this way. The bulk of the assessment activity for business simulations and games has focused on this internal validity” question. As several reviews have stated [7. 11, 13] little conclusive evidence has been gleaned from these activities. The second point of assessment occurs after the trainee has gone into the work environment. This assessment is usually termed “external validity.” Little effort seems to have been put forth in assessing “external validity” (see [12] for an exception) probably due to the difficulties involved in a longitudinal study of this type. In summary, the results of assessment activities for business simulations provides little support for their use as training methods. Thus, with this assessment, two hypotheses ire possible: 1) business simulations are not useful tools in management education, or 2) there are deficiencies in the application of the training program construction process to business simulations, Before the first hypothesis can be accepted, the second must be evaluated from t methodologically rigorous standpoint. EXPERIMENTATION Efforts to more rigorously evaluate the educational value of business games requires in experimentation approach in contrast to exploratory or descriptive methods. Kerlinger [81 views in experiment as: …a scientific investigation in which an investigator manipulates and controls one or more independent variables and observes the dependent variable or variables for variation concomitant to the manipulation of the independent variables. Thus, the critical components of the experimental approach would appear to be: I) determination of the independent variables and dependent variables, 2) control of the independent variables, 3) manipulation of the independent variable or variables of interest (experimental variable/s) and 4) observation of the dependent variable or variables (criterion variable/s). At the “external validation" point of assessment, the dependent variable or variables would be those variables which define job success. At the “internal validation” assessment point, the dependent variables should be those which link with the job success variables. These dependent variables should be readily apparent in the training goals. Parasuraman [13] has stated that the implied goal of business simulations is to teach effective decision making. Thus, a prime candidate for the ‘internal validation criterion or dependent variable should be decision making ability. Independent (experimental) variables in the business simulation assessment experiment would be those which have an impact on the dependent variable or variables and can be manipulated by the experimenter. These would be variables involved with the game itself. Some of the variables proposed by Schellenberger include: degree of reality, level of complexity, mode of administration, intensity and duration of play, administrator intervention, and instructor attitudes [14]. Note that not all variables that could have an impact on the criterion variable (decision making ability) ire considered as potential experimental variables. Those variables which can not be manipulated by the experimenter, such as basic intelligence, traits, and values, must be controlled. Without control, it is impossible to show causality in the assessment experiment. The final task in the business simulation assessment experiment is observation or measurement of the dependent or criterion variable or variables. Thus, some method of measuring decision making abilities in the “internal validation” assessment or job success in the °external validation assessment point is necessary. Also of importance in this step, particularly for the “external validation” point, is time of observation or measurement. How long does it take for the individual’s characteristics and the work environment factors to interact to produce some level of job success? A PROPOSED APPLICATION Using the training program paradigm as a basis, t proposed experiment for assessing the value of business simulations will be presented. Although the major aim of business education is to increase the chances of job success in “external validation’ assessment task, the training program (business simulation) must be assessed first. If this “internal validity” assessment is not completed first, it would be impossible to determine if a negative ‘external validity” assessment finding was due to lack of acquiring the skills sought by the training program or the inability of the skills to be transferred to the work place. Thus, this proposed experiment is of the “internal validity" assessment type. 7 Developments in Business Simulation & Experiential Exercises, Volume 10, 1983 The experimental variable of interest in the proposed experiment is administrator intervention. Administrative intervention refers to all actions taken by the instructor to reinforce the learning goals. The Intervention variable will be manipulated by having a low intervention treatment and a high intervention treatment. The low intervention treatment will consist of a taped program lecture, trial decision, out-of- class central consultants, and a post game quiz. The high intervention will include the above activities plus a pre game quiz, required minimum purchases of market research, board of directors meetings, planning documents, sales and market share forecasting consul- tints, and post game critiques. The general hypothesis of the proposed study is that high intervention will result in higher levels of the criterion variable-decision making ability, than the low intervention treatment. Other independent variables will be controlled in the following ways. Student groups will be matched on the basis of age, sex, major (accounting and non- accounting) and grade point average. Basic ability, traits, and values will be measured using standard instruments. Groups will be evaluated to determine if there are any significant differences between the treatments on these variables. The potential problem of hiving the nature of one particular business simulation as a training method will be controlled by using two complex games. The experimental design can then be represented by the following four cell matrix (Figure 3). A final possible m control this variabl with low interventio intervention (using will compete only w As with most res interest, motivation will be measured. concepts will be as dependent variable making ability. Bo developed and used decision making ab the major criterion v the end of the term. the hypothesis that simulations will ha those in the low inte REFERENCES [1] Boseman, Glen and Robert E. Schellenberger, ‘Business Gaming: An Empirical Appraisal," Simulation and Games, Vol. 5, No. 4, 1974, pp. 383.402. [2] Campbell, J. P. “Personal Training and Development,” Annual Review of Psychology, 1971. [3] Campbell, J. P., Dunnette, M. D. et al. Managerial Behavior Performance and Effectiveness. 1970, McGraw-Hill. [4] Gagne, R. M. “Military Training and Principles of Learning," American Psychologist, 1962, Vol.17 pp. 83-91. [5] Ghiselli, E. C. Explorations in Managerial Talent. 1971, Goodyear Publishing Company. [6] Goldstein, Irwin J. Training: Program Development and Evaluation. (Monterey, California: Brooks/Cole, 1974). [7] Greenlaw, Paul S. and F. Pail Wyman, ‘The Teaching Effectiveness of Games in Collegiate Business Courses,” Simulation and Games, Vol. 4, No. 3, 1973, pp. 259-294. FIGURE 3 EXPERIMENTAL DESIGN [8] Kerlinger, Fred N. Foundations of Behavioral Research. (New York: Holt, Rhinehart and Winston, 1973), p. 315. [9] Mahoney, T. E. and Jerdee, T. H. et al, “Predicting Intervention Game A Game B High Low 8 oderating variable is the Instructor. To e, each instructor will have one section n (using both games) and one with high both games). As a final control, groups ith like Intervention groups. earch on management games, student , attitudes and perceptions of learning Also, their understanding of certain sessed. However, the major criterion or of this proposed project will be decision seman and Schellenberger [15] have seven! short structured cases to assess ility. These cases will be used to measure ariable Measurements will be taken near Analysis of variance will be used to test high intervention groups in both business ve better decision making abilities than rvention groups. Managerial Effectiveness,” Personnel Psychology, 1960, 13, 81-98. [10] Mahoney, T. E. and Jerdee, T. H. et al, Development of Managerial Performance: A Research Approach. Southwestern, 1963. [11] Neuhauser, John J, , “Business Games Have Failed,” Academy of Management Review, Vol. 1, 1976, pp. 124-128. [12] Norris, Swight R., and Charles A. Snyder, “External Validation: An Experimental Approach to Determining the Worth of Simulation Games,” in Proceedings, Eighth ABSEL Conference, 1981, pp. 24 7-250. [13] Parasuraman, A., “Assessing the Worth of Business Games,” Simulation and Games, Vol. 12, No. 2, 1981, pp. 189-200. [14] Schellenberger, Robert E. , “Critical Variables in Research on the Educational Value of Management Games,” Proceedings, Eighth ABSEL Conference, 1981, p. 70. [15] Schellenberger, Robert E. and Boseman, F. G. Policy Formulation and Strategy Management, 1981, 2nd Edition, John Wiley and Sons. Table of Contents Volume 10, 1983 Is the Computerized Business Simulation Relevant? Business Professionals Play a Student Game A Methodology For Assessing the Internal Validity of Business Simulations A Longitudinal Study of the External Validity of a Business Management Game Policy Analysis and Decision: A corporate Relocation Simulation Exercise Effective Listening: An Exercise in Managerial Communication The Symbol Exercise: An Initial Group Activity Concept Based Simulations Institutional Users of Experiential Learning Packages: A Preliminary View from Publishers' Adoption Lists How to Use Business Games in the Business Policy Course: The Students' Perspective Professors' Ratings of Business Policy Learning Methods Moot Trial: An Exercise in Trial Procedure and Evidence The Advertising Agency Game: An Experiential learning Exercise The Use of Videotaped Cases in Teaching Information Acquisition and Decision-Making Skills Experiencing Information Processing Strategies as a Means to Explore Decision Making Importance Ratings and Operations Data as Predictors of Business Game Performance Determinants of Performance in Computer Simulations Predicting Business Game Performance form Perceptions of Manager Information and actions Business Consulting: A Practicum for Undergraduate Internship as a Contingency Based Experiential Learning Program for More Effective Organizational Socialization: A Conceptual Framework MANSYM III Decision Support System Demonstration Learning the Concept of market Value Through Simulation Conflict Management for Economic Developers Development of Data Analysis Units Designed to Enhance Reasoning and Knowledge Transfer in the College Level Course BOSS: A Behavioral-Quantitative, Computer-Supported Game Entrepreneurial Potential: An Experiential Exercise in Self Analysis and Group Assessment Development of Strategists: Simulated Cases BANKRUPT: A Deceptively Simple Business Strategy Game Simulating Market and Firm Level Demand - A Robust Demand System COMPSIM A Computer Center Management Simulation Hiving Model: Assessing Management Skill Awareness The Johari Window, A Reconceptualization Role-Playing Based on Video-Tape Scenarios: An Application of Modeling to Building Supervisory Skills How to Internationalize Your Curriculum A Computerized Model of Human Behavior in a Total-Firm Management Simulation the Worksheet Approach for Simulation Game Strategy Analysis Teaching Competitive Bidding Using a DSS Generator Do We Learn from Experience? The Use of Theory Power for Increased Research Momentum in Business Simulation and Experiential Exercises Research Report on Programmatic Research on Perceived Learning Barriers with Simulation and Experiential learning An Empirical Examination of Conflict - and Nonconflict - Oriented Problem-Solving Technologies Management Curriculum: 1982 Humor as a Management Tool: Use in Formal Game Presentations Student Behavioral Change Through Teacher behavioral Change