EXTERNAL VALIDATION: AN EXPERIMENTAL APPROACH TO DETERMINING THE WORTH OF SIMULATION GAMES Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 247 EXTERNAL VALIDATION: AN EXPERIMENTAL APPROACH TO DETERMINING THE WORTH OF SIMULATION GAMES Dwight R. Norris, Auburn University Charles A. Snyder, Auburn University ABSTRACT Most research designed to measure the worth of simulation games as pedagogical devices has examined the internal validity of games--their value in teaching specific material to players. This paper reports a pilot study designed to measure the external validity of games--the transferability of game learning to real-world applications. Results failed to support a significant relationship between game performance success and three measures of subsequent career success. INTRODUCTION Since the development of simulation games more than two decades ago, substantial progress has been made in evaluating the worth of these games. The preponderance of published research has examined the internal validity of games--that is, the pedagogical value of simulations in teaching specific material to participants. Very few studies have been reported which examine the external validity of games--the transferability of academic insights into useful and effective real-world orientations, perceptions and business career practices [12] . This paper reports a pilot study conducted by the authors which longitudinally analyzes the external validity of a widely used general management game. We examined the performance success of a group of students playing a management game, and compared these results with measures of career success taken five years after graduation. IMPORTANCE OF EXTERNAL VALIDATION Shortly after the appearance of the first management games, Hubbard recognized the importance of external validation when he expressed an opinion that "the real test of the validity of management gaming as a teaching technique is how it actually affects the students when they do get out into the real world of business” [6, p. 30]. External validation is germane to important evaluative questions concerning the worth of simulation games. For example, does gaming experience enable the participants to transfer academic insights gained in play to useful and effective real world career practices? As Parasuraman [7] argues, external validation would enable game users to determine if simulations have any practical value to participants after they leave a course. Additionally, external validation would either support or reject the contention that complex computer-based simulations can serve as laboratories for studying organizational problems. RESEARCH EXAMINING EXTERNAL VALIDITY Without exception, the only studies reported in published research which are relevant to external validity compare the relative performance of businessmen and students playing a management game. Three studies were located. Babb et al. [1] report a series of studies examining the success of businessmen compared with students in game performance. Businessmen were found to be more stable and conservative than a contrasting group of students. The businessmen relied heavily on their business experience, while students relied more on the computer output as a basis for their decisions. The authors conclude that students behaved somewhat differently in game play than the businessmen owing to the students’ temperament, experience, age, and preconditioning. Vance and Gray [8] studied industry executives and students playing a business policy game. They reported a high degree of trait similarity between those students and executives who were successful in playing the same business game. The two groups were found to be similar in self-assurance, decision- making ability, perceived occupational level and initiative. Wolfe [12] used business school students and practicing businessmen to examine the correlation between real- world career success and predicted game success. In addition to differences in decision-making mechanics and use of decision-making aids, Wolfe reported a moderate to high positive correlation between the businessmen’s relative real world success and the relative success they individually and jointly achieved when operating in their simulated environment. In each of these studies, the authors fail to clearly state the relationship between their studies and the external validity of the games used. Apparently the studies seek to demonstrate that successful, experienced businessmen achieve higher levels of game performance than inexperienced students. Such a demonstration is apparently offered as evidence that simulation games are relevant to the real-world and that skills required to successfully perform simulated decision making are the same skills necessary to achieve career success. As Wolfe [12] suggests, a more appropriate test of external validity would be a longitudinal study in which game performance is compared with later career success for the same Individuals. Since simulation implies that a model is employed to parallel reality, this type of study would enable the researcher to draw conclusions about relevant aspects of the two environments. In the case of a business simulation, a significant correlation between the student’s game performance and subsequent performance in the real environment would constitute viable evidence as to the external validity of the game. MEASURES OF CAREER SUCCESS In the present study, career success is employed as a criterion; therefore, it is useful to explore the definitions of career effectiveness. There is no real agreement on what should be included in the concept of career success. In a review of the research literature, Hall (4] found that career effectiveness has been generally defined in terms of performance as well as money and position. In this sense, some of the measures of career success center around rewards; that is, if an individual meets performance expectations, he or she is Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 248 usually rewarded (2] . The wide variety of rewards, coupled with their varying meanings to individuals, tends to complicate their use in measuring career success. Performance A frequently employed measure of effectiveness is the supervisory rating of performance. The widespread use of performance appraisals has created a sizable data base, albeit a profoundly inconsistent one. One of the greatest obstacles to the researchers has been imposed by the need to maintain confidentiality of the performance appraisals. Some additional performance indicators are supervisory ratings of success, contributions to the organization, number of subordinates, budget size, and earnings of the individual’s organizational unit [4] Monetary Reward Salary levels, average annual salary increases, and salary comparisons with peers provide some typical financial indicators of career success. Fringe benefits, stock-options, and other supplemental payments may be incorporated in the financial reward category. Although usually perceived as non-monetary, perquisites might be categorized and added to salary levels as a form of remuneration. Position Position refers to the rank or level in the organizational hierarchy, number of promotions received over a given period of time, or length of time in present job [4]. . Job title is sometimes used to determine rank or success. However, the policy in certain organizations of creating titles renders this a suspect measure. Difficulties in Measurements Relative career success measurement is complicated by the different criteria. Because of the lack of a consistent measure of subjective criteria such as approval or praise, it is difficult to precisely indicate significant variations in success when these criteria are employed. Using promotions or salary increases as indices of career success is considerably better. However, these criteria are also complicated by intercompany and intracompany transfers, organizational differences, industry factors, external economic conditions, confidentiality of information (an especially thorny issue in obtaining access to performance appraisals), having to rely to a great extent upon self- reports with the attendant risk of bias, and difficulty in designing a valid research instrument. HYPOTHESIS Of course, no simulation can tap all the characteristics or abilities required for a successful career. However, to the extent that the simulation game under study parallels reality, one would expect that the characteristics and abilities required to achieve career success might be similar to those characteristics and abilities required for game success. A review of the literature examining game validity offers sufficient evidence to hypothesize that those students who achieve higher levels of game success will also achieve higher levels of career success. A full analysis of the determinants of game success and the determinants of career success is beyond the scope of this paper. Such an analysis would logically follow the finding of a relationship between game success and career success as the experimenter attempts to determine common variants between the two success measures. However, two variables (grade point average (CPA) and organization size) were analyzed to assess their influence on game and career success. METHODOLOGY A survey instrument was used to collect relevant career success data from graduates of the school of business of a large university. Three responses were used as measures of the career success subjects had achieved since graduation. These include: (1) the number of promotions the subject had received either within the same organization or into another organization since graduation. (2) The number of organization levels that exist between the subjects’ chief executive officer and their present position (the fewer the number of levels, the greater the career success) (3) The percentage salary increase subjects had received since assuming their first position after graduation. Subjects indicated their response on a Likert type scale. A total of 100 questionnaires were mailed, with 54 returned for a response rate, after a follow-up letter, of 54 percent. Subjects surveyed had played a general management game [5] in the fall, winter or spring quarter five years prior to the survey. The game was a part of the capstone course in business policy. The same instructor taught each course and provided all evaluations. Came success was measured by a combination of ending return on investment (ROI) and general game participation with ROI being the dominant variable, Subjects were assigned a game grade based on a combination of these two measures. To insure the comparability of game performance across different business policy classes, the subjects’ game performance was ranked from 1 to 5, with a rank of 1 representing the lowest performance and a rank of 5 the highest level of performance. RESULTS To assess the reliability of the three success measures, subjects were asked to indicate their satisfaction with career progress and their satisfaction with salary increases since graduation. Table 1 reports the Pearson correlation coefficients between the three measures of career success and the two satisfaction measures. Of Table 1 Zero Order Correlations Between the Three Measures of Career Success, Satisfaction with Career Progress and Satisfaction with Salary Increases Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 249 the three career success measures, only organization levels and salary increases were related to one another. Supporting the reliability of these measures, 8ubjects who had received a greater number of promotions, and a higher percentage of salary increases were significantly more satisfied with their career progress and their salary increases. Though the number of organization levels was positively related to the satisfaction levels, the correlations were not significant. To examine the relationship between the measures of career success and performance in playing the simulation game, analysis of variance (ANOVA) was conducted. Using the five rankings of game performance results as independent factors, promotions, organization levels and salary increases (career success measures) were analyzed under ANOVA as dependent variables. These results are reported in Table 2. The ANOVA results did not reveal significant differences between the game performance ranks and later promotions, organization levels or salary increases. In fact, the only variation between means of performance ranks occurred on the number of organization levels between the respondent and the employing CEO, though this variation was not significant. These results do not support the hypothesis that subjects who perform at a high level on simulation games also achieve higher levels of career success. The literature examining correlates with game performance suggests that CPA may be an important determinant of game play, although the literature is mixed (8; 3] . GPA might also be expected to influence career success, although again the literature is inconclusive [11]. . Additionally, organization size (measured by number of employees) might be expected to influence certain career success measures, perhaps the number of promotions received and the number of organization levels between the respondent and the CEO. To assess the influence these characteristics have on game performance and career success, correlations were determined. These results are reported in Table 3. As suggested by a number of gaming studies, CPA was significantly related to game performance success. GPA was also related to the organization levels measure of career success and both personal satisfaction measures. Organization size was related to two career success measures, but in opposite directions. Organization size was not related to prior game performance nor to personal satisfaction measures. DISCUSSION The results of this study lend little support for the hypothesis that success in a general management simulation is related to subsequent career success. However, the study does indicate a number of avenues for further research examining the external validity of simulation games. First, the factors which contribute to success in a general business simulation may be inappropriate in determining the career success of graduates five years after graduation. Although empirical evidence is lacking, there is a contention that business schools and simulation games train individuals to view management from a long-range or top management perspective. However, graduates enter the business world at levels with much shorter-range perspectives (see, for example, “What are they teaching.. .“ and “Broadening the B- Schools,” Business Week, Nov. 10, 1980). It is possible that the subjects of this study are in shorter-range jobs and have not been able to demonstrate, in their careers, the characteristics that led to their successful game performance. Career success measures taken at later career stages may produce quite different results. Second, the two moderating variables analyzed (GPA and organization size) appear to have influenced the results of the study. GPA seems to influence both game and career success, though an analysis of the two success measures, statistically controlling for CPA, did not detect a significant relationship. Future studies of these relationships are warranted, with an awareness of the influence CPA may have on both success measures in mind. Additionally, organization size appears to have an effect on two of the three career success measures. As would be expected, subjects in larger organizations were more distant from the CEO. However, contrary to expectations, persons in larger organizations reported fewer promotions. Regardless of the explanations, organization size does appear to be an influential variable. Third, future studies might place subjects in taxonomic categories on the basis of the information received. The small sample size resulting from such segmentation would not be appropriate for the present study, but should be considered for any follow-up investigation. Fourth, an attempt should be made to obtain supplemental indicators of career success through performance appraisals, supervisor ratings, etc. The use of such ratings or assessments could not be incorporated in this study; however, they should be valuable in future studies. Finally, follow-up studies might include: (1) specific questions on bonuses or other special incentive awards; (2) an assessment by respondents comparing their raise data and promotion data with others in their organization; a categorization of respondents by functional area such as technical, financial, personnel, etc. ; and Developments in Business Simulation & Experiential Exercises, Volume 8, 1981 250 an approximation or measure of the results of the respondents’ organizational unit such as ROI or percent improvement etc. Despite the lack of support for the hypothesis in the present study, the results should not discourage further efforts to investigate the external validity of simulation games. Further research is indicated to incorporate the preceding suggestions for improvement of future studies. Only if vigorous efforts are expended in such assessments will the users of simulation games be able to point to a record of empirical effectiveness. REFERENCES [1] Babb, E. M., M. A. Leslie, and N. D. Van Slyke. ‘The Potential of Business Gaming Methods in Research,’ The Journal of Business, Vol. 39, 1966, pp. 465-472. [2] Berlew, D. E. and D. T. Hall. “The Socialization of Managers: Effects of Expectations on Performance,” Administrative Science Quarterly, Vol. 1]., 1966, pp. 207-223. [3] Gray, C. L., Performance as a Criterion Variable in Measuring Business Game Success: An Experiment with a Multiple Objective Performance Model,’ in Decision Sciences in Instruction and Application, The Proceedings of the Southeastern American Institute of Decision Sciences, Memphis, TN, 1972, pp. 193-198. [4] Hall, D. T., Careers in Organizations (Pacific Palisades, CA: Goodyear, 1976). [5] Henshaw, R. C., Jr. and J. R. Jackson, The Executive Game (Homewood, IL: Richard D. Irwin, 1972). [6] Hubbard, C. L. as quoted in Albert Newgarden (editor), Simulation and Gaming: A Symposium (New York: American Management Association, 1966). [7] Parasuraman, A. “Evaluation of Simulation Games: A Critical Look at Past Efforts and Future Needs,” Proceedings, Seventh ABSEL Conference, 1980, pp. 192-194. [8] Vance, S. C. and C. F. Gray. ‘Use of Performance Evaluation Model for Research in Business Gaming," Academy of Management Journal, Vol. 10, 1967, pp. 27-37. [9] Walczak, L. (Ed.). “Broadening the B-Schools,” Business Week, November 10, 1980, p. 158. [10] What Are They Teaching in the B-Schools?” Business Week, November 10, 1980, pp. 61-69. [11] Williams, F. J. and T. W. Harrell. “Predicting Success in Business,” Journal of Applied Psychology, Vol. 48, 1964, pp. 164-167. [12] Wolfe, J. “Correlates and Measures of the External Validity of Computer-based Business Policy Decision- Making Environments,” Simulation and Games, Vol. 7, 1976, pp. 411-438. Table of Contents Volume 8, 1981 The Promotion: Human Sexuality in Organization The Simulation of Chaos Leadership Development in a Simulated Urban/Suburban (U/S) Environment Tomed: A Computer Game Emphasizing Social Responsibility/or/why the Pop-Top Can? Integrated Brain Activity and the Manager's Job: Utilizing the Troika Model What does R2 Have to do with a Product Management Course? An Analysis of the Effects of Jungian Problem-Solving Style Dimensions on Marketing Decisions Bargaining Behavior in Personal Selling and Buying Exchanges Extending the Simulation Product Life Cycle A Generalized Algorithm for Designing and Developing Business Simulations Operationalizing a Test of a Model of the Use of Simulation Games and Experiential Exercises Pygmalion and Perception: An Experiential Exercise Behavioral Consequences of Reward Regarding Employee Absenteeism in an Industrial Setting: An Operant Conditioning Approach The Simlab Program: The Use of Experimental Simulation and Process Analysis for the Development of Management and Organizations Designs for Research on Simulation-Games, Cases, and Other Experiential Exercises The Role of Students in The Case Method Weaknesses in Research Design Critical Variables in Research on the Educational Value of Management Games Research Questions for Cases Research on the Learning Effectiveness of Business Simulation Games - A Review of the State of the Science The Effects of Valuation Techniques on Holding Cost During Inflationary Periods: A Simulation Exercise The Operations Simulation - A Study in Game Development Applying Guided Design to the Production/Operations Management Course: A Progress Report and Evaluation Terminal Data Entry and Retrieval Systems Simulations and Microprocessors Microcomputers and Related Technology for Simulation Gaming Microcomputers - A New Technology for Innovations in Business Simulations Microprocessor Controlled Interactive Video Simulation Business Game Design: From Theory to Practice The Success of a Computerized Simulation in Microeconomic Pedagogy The Test Preview Game: Applying the Game Show Format Providing a Real World View of the Personal Function: A Simulation Finding an Effective Means of Teaching Managerial Behavioral Skills: Two Different Experiential Teaching Methods Compared A Management Development Program Based on the Experiential Learning Model Participant Type Differences in Response to Experiential Methods: An Informal Look Preparing Student Groups to Participate in Experiential Group Projects: An Organizational Development Approach An Instrument for the Assessment of Learning Dimensions: A Progress Report of the Learning Dimension Scale (LDS) An Empirical Analysis Relating the Learning Style Inventory to Memory and Logical Ability Teaching Styles in Simulation Experiential Learning Versus Traditional Teaching Styles Student Perceptions of Effective Teaching Behaviors Problems in Evaluation of Experiential Learning in Management Education Students' Perceptions of Learning by Simulation A Relative Evaluation of Experiential and Simulation Learning in Terms of Perceptions of Effected Changes in Students Overview of Computer Based Business Games in Business Policy Classes Behavioral Decision Theory and Business Policy Giving Accounting Students Writing Experience as Job Preparation Getting to First Base with MBO: An exercise for Writing and Evaluating Objectives Dimensions of Conflict in Experiential Learning Consumer Alienation and Perceived Relevance of the Business Simulation Using the Self-Reference Criterion to Simulate Culture in Internationalized Business Course Experiential Learning in a Cross Cultural Setting- The Practice of Simulation Approach to Business Education in Japanese Universities Student' Perceptions of the Use of a Computerized Simulation in Teaching Management Information Systems Using the Case Study Approach to Develop a Microcomputer Bases, Fully Integrated, Data Base Driven, Management Information System The Case Study as a Tool for Organizational Change: Applying the Steel Ax to the Designers of Management Information Systems Toward a Theory of Teaching Business Policy A Case Study in the Use of Experiential Learning (A Management Game Simulation) to Enhance Student Understanding of Strategy Evaluation and Policy Formulation Suggestions for Integration of the Business Administration Core Publishing Opportunities and Requirements for Business Simulation and Experiential Learning Materials How do we Apply Experiential Learning Intercollegiate Case Competitions for M.B.A. Students: Initiation and Implementation Teaching Business Policy and Strategy Using the Incident Process The Learning Co-Op Approach to the Core Policy Course Meeting the Managerial Skill Shortage - Is Academia Up to It? An Empirical Analysis of Experiential Learning Reinforcement International Experiential Learning: Experience is the Best Teacher Encouraging Student Participation During International Academic Programs European Summer Study Program: Can you, Should you, What Does it Take? Travel Seminars in Europe: How can I Direct One? International Experiential Learning: Student Evaluation ABSEL: Empirical Findings on the State of the Association Sensitivity of Performance Scores in Business Simulations Improving the Learning of a Business Simulation Game by Increasing the Process Content Student Participation in Deciding Performance Criteria for Grading in an OB Course: An Exercise and a Case Study Markup for Profit: A Simulated Self-Administered Experience in Retail Pricing The Investment Decision Game: An Experiential Learning Approach to Stock Market Decisions Through Gaming CHIPS: A Marketing Channels management Game External Validation: An Experimental Approach to Determining the Worth of Simulation Games Teaching Performance Appraisal Skills: An Experiential Approach In Support of Experiential learning: Results of a Follow-Up Survey The Introductory Management Course: Taking Theory Application One Step Further Decision Efficiency and Effectiveness in a Business Simulation The Implications of Cognitive Processing Variables and the Complex Decision Simulating the Simulation for Enhanced Player Rationality Simulation/Experiential Learning Audit