EXTERNAL VALIDITY OF BUSINESS GAMES Developments in Business Simulation & Experiential Exercises, Volume 13, 1986 126 External Validity of Business Games Dwight R. Norris, Auburn University ABSTRACT The pedagogical value of games has received considerable research attention in the gaming literature. However, most of this research focuses on the educational value of games -- that is, their internal validity as teaching devices. This paper examines the concept of external validity as associated with business games. A contrast between internal and external validity is drawn, together with a review of differing definitions of external validity. Research examining external validation is reviewed. Finally, suggestions and recommendations are offered to guide future validation research. INTRODUCTION For approximately three decades, business games have been used in educational settings to provide experiential learning tar business students. The interest shown by academicians in using simulation games as pedagogical tools in collegiate courses has been gaining strength during the past few years [17]. In increasing numbers, games have been made a regular part of the academic curriculum. The sustained interest in business games as teaching tools has fostered research efforts to determine the effectiveness of games. Substantial progress has been made in evaluating the pedagogical value of such games. The preponderance of published research has examined the internal validity of games--that is, the educational value of simulations in teaching specific material to participants [3; 7; 18; 21]. In contrast, few studies have been reported that examine the external validity of games. Perhaps a reason for limited study is a failure by researchers to agree on the meaning of external validity. Whereas something ot a consensus has developed regarding the meaning of internal validity (i.e., teaching effectiveness) of business simulations, a number of differing concepts of external validity have emerged in the gaming literature. One concept of external validity is the degree to which the game corresponds with the real life situation (an external phenomenon) it is intended to represent. In other words, is the game a valid representation of external reality? This concept is often termed realism in the gaming literature. However, little consensus exists on the proper term for this type of validity (or realism). Mihram [14] defines a term which he calls “event validity’ as being established by a comparison of the predictive responses of the model with past recorded history of the actual system or phenomenon. This seems very similar to realism or validity as expressed above. Similarly, Boocock [2] uses the term “empirical validity’ to refer to the closeness of fit of the game structure or outcomes to other measures serving as criteria of the phenomena under study. Dukes and WaIler [5] use the term “accuracy” to refer to how closely the game mirrors the model of reality it represents. Both terms are very similar to the first concept of validity (realism). Since the management game represents an abstraction from some real process, and hence only an approximation, it is fundamental that the game should appear to be realistic and behave in a realistic fashion. But realism adds complexity, and as the game model becomes more complex, the game becomes more difficult to learn, to play, and to administer. Too complex a game model does not permit identification of the impact of important policies and decisions and isolation of underlying structured relationships and causal factors. Therefore, the designer is often faced with a basic dilemma: the conflict between realism and simplicity of computation and play. Realism is often quite measurable in scientific simulation models which have the major purpose of problem-solving or research involving the modeled environment. Indeed, relatively simple validation processes for scientific models have been described in the gaming literature [9; 13; 14]. However, it became apparent very early in management gaming that the process of validation of scientific models was very difficult if not impossible to apply to management games [9]. Most games were not designed to reflect reality in a scientific manner, nor to be predictive or descriptive of the real world it sought to represent. Unlike scientific processes, business processes are affected by literally hundreds of variables, many of which are qualitative and beyond the scope of mathematical simulation. External validation through assessment of the realism of the business game model in the sense a scientific model might be evaluated has met with little success. Further discouraging efforts to evaluate game models “scientifically” are assertions regarding gaming verisimilitude. In perhaps the first succinct discussion of this second concept of validity in management gaming, Kibbee [ii] notes that most games are designed as training devices, and the object of the game is not to study or mathematically replicate a specific problem or situation, but to evidence general principles. Therefore the validity of the game model- -that is, the degree to which it corresponds with reality--is only of minor importance. Kibbee notes that, “What is far more important in most management games is verisimilitude: the degree to which the players feel that the simulated situation is real” [ii, p. 9]. Boocock uses plausibility” in the place of verisimilitude. He notes that, “Plausibility may be relatively more important [than scientific validation] in gaming because it is posited upon capturing participants’ interest and involvement so that they behave in the simulated environment as they do or would in real life” [2, p. 33]. Within gaming literature, verisimilitude has been referred to as face validity [14], or as common sense or logical validity [11]. Kibbee notes that in one sense realism and verisimilitude are not in conflict, indeed, “The most obvious way to achieve verisimilitude, of course, is to build a realistic model” [11, p. 9]. But Meier et al. [13] note that while verisimilitude can be achieved with a realistic model, it may also be achieved with a simple model if it is properly structured. Herein lies the conflict cited above, for increased realism often Developments in Business Simulation & Experiential Exercises, Volume 13, 1986 127 has higher costs and adds to the complexity of the model, running the risk that the perceived relationships between player decisions and game performance criteria will be less visible to participants. Most management games rely heavily on the creation of an illusion, and the designer must decide if the purpose for which the game is to be used necessitates the added realism, and its added cost and complexity. Kibbee points out that cost and complexity may not be the only conflict between realism and verisimilitude: …sometimes it may even be necessary to make certain parts of a model intentionally unrealistic in order for the whole to seem realistic to the players. This aspect of verisimilitude is perhaps most familiar to us in the theater, where a true representation of reality may be far less convincing than an exaggerated or simplified representation. [11, p. 9] Additionally, Greenlaw, Herron, and Rawdon illustrate another reason for conflict between realism and verisimilitude: .[I]t may well be that one of the major values of games is that they do not provide completely accurate representation of the real world. Reality is cluttered with the insignificant and the trivial as well as that which we consider important. By distilling from highly complex real world situation only chose decision factors and organizational processes which are considered most critical to managerial effectiveness, the designer can focus the attention of participants on those facets of management where he believes the greatest need for understanding exists. [6, pp. 78-79] From this analysis, one might conclude that to a point, increased realism enhances verisimilitude, but may exhibit decreasing returns in terms of verisimilitude beyond this point. But in noting a relationship between the two concepts of external validity, it must not be forgotten that their basic nature is quite distinct. Where realism is a measure of the actual agreement between a model and a modeled situation, verisimilitude is a measure of the perceptual agreement between a modeled role and the actual real world role. Actual agreement, realism, may almost always be enhanced by increased complexity--adding more elements or relationships to the model. But whether increased complexity enhances the illusion or perception of reality is a separate matter. The measurement of realism may be quite readily accomplished, but the measurement of verisimilitude is much more complex and difficult. From a preponderance of research assessing the teaching effectiveness of business games emerges a third concept of external validity--the assessment of successful game attitudes, abilities, and behaviors relative to successful business career characteristics. Hubbard [9] feels that “the real test of the validity of management gaming as a teaching technique is how it actually affects the students when they do get out into the real world of business Kossach concurs, citing the real issue as “whether--and if so, in what way-- gaming experience changes the performance of the players when they leave the game situation and engage in real-life business decision making’ [12, p. 9]. Wolfe [20] actually uses the term “external validity” to refer to the transferability of academic insights into useful and effective real-world orientations, perceptions, and business career practices. Supporting this concept of external validity, limited research has emerged addressing the question of whether the gaming experience requires characteristics and abilities for successful play that are similar to those necessary for success in a real-world business career. Parasuraman [17] is critical of past evaluation efforts that have tailed to ascertain whether games will have any practical value to students after they leave a course. Such external validation would enable game users to determine if simulations have practical value for predicting future career success. Moreover, external validation would lend to support or reject the contention that. complex computer-based simulations can serve as laboratories for studying organizational problems. Research examining this latter form of external validity is reviewed below. RESEARCH EXAMINING EXTERNAL VALIDITY External validation studies that compare business game success to business career success range from simple to quasi-experimental studies. For example, Kaufman [10] surveyed a group of alumni regarding their perceptions of the extent to which a business game played in their college curriculum was later helpful on their job. Results were very supportive of game use. Byrne [4] investigated the extent to which a business game allowed students to engage in ten common managerial roles and skills identified by Mintzberg [15]. It was reasoned that opportunity to engage in such roles and skills would be beneficial to later business careers. Students were found to have performed eight of ten roles over an extended period of game play. A group of more complex studies have assessed external validity by comparing game performance of businessmen with performance of a group of students. Babb, Leslie, and Slyke [1] report a series of studies examining the game success of businessmen versus business students. Businessmen were found to be more stable and conservative than a contrasting group of students. The businessmen relied heavily on their experience, while students relied more heavily on computer output as a basis for their decisions. These authors concluded chat students behave somewhat differently in game play than businessmen owing to the students temperament, experience, age, and preconditioning. Vance and Gray [19] also studied business executives and students participating in a business policy game. A high degree of trait similarity between students and executives who were successful in playing the same business game was noted. Similar traits included self-assurance, decision- making ability, perceived occupational level, and initiative. In a third study using seasoned middle managers, Wolfe [20] used a quasi-longitudinal design to compare the businessmen’s game success with their real- world career success. An index of upward career mobility was created based on their self-reported ten- year salary and promotion histories. Those with similar career mobility indices were assigned to three-person teams. Results showed a strong correlation (r = .75) between the team’s career index and its economic performance. Wolfe concluded that the game had external validity since the executives who experienced the most promotions, developmental moves, and higher salaries performed better in the same simulation. Developments in Business Simulation & Experiential Exercises, Volume 13, 1986 128 Though this group of studies provide evidence that success in business game play requires abilities needed for success in a business career, Wolfe [20] suggests more appropriate tests of this form of external validity are those using a longitudinal design in which game performance is compared with later career success for the same group of individuals. Two more recent studies have adopted this approach to more completely address external validity. Norris and Snyder [16] surveyed 100 business policy students some five years after graduation (response rate was 54 percent). Each student had played a general management game [8] in their senior college year. Game success was measured primarily by ranking ending return on investment from I to 5 (1 represents low game performance). Completed mailed questionnaires gathered self-report data on promotions, proximity to their organization’s chief executive officer, salary increases, and salary and career satisfaction. When career success measures were compared with game success measures, no significant relationships were found. Wolfe and Roberts [22] extended the Norris and Synder [16] study by comparing career success measures with single- member game performance. The latter study had used multiple-member firms. A total of 142 business policy seniors were tracked for five years after graduation via telephone interviews and mailed questionnaires. Their self- reported career success was compared to their in-class game performance. Career success was treated globally in the form of salary levels, salary changes, promotions, and job satisfaction. Game success was measured by earnings and return on investment and return on equity. Game performance was positively associated with absolute salary levels (r = .267, p < .001) and job satisfaction (r = .191, p < .05) but unrelated to salary changes or promotions. A predictive step- wise multiple regression model found that at least one game-related feature was associated with each career success measure, although those features were never the most important predictors. DISCUSSION The concept of validity in business games has both an internal and external basis. Whereas internal validity has evolved to refer primarily to teaching effectiveness, considerable disagreement occurs over the meaning of external validity. External validity in the form of realism emerged as an early concern in the gaming literature. Later, the concept of verisimilitude (i.e., the perception of realism) was felt to be most important. Demonstrating external validity in the form of realism or verisimilitude proved most difficult, since each concept is largely ‘in the eye of the beholder.” More recently, researchers have turned to studies of game success versus career success as an alternative test of external validity. Of interest is the degree to which characteristics and abilities required for successful play of business games are also required in a successful career, in an indirect way, this question taps realism and verisimilitude. A realistic game model and format should demand of the student characteristics and abilities required in a business career. The perception of realism (verisimilitude) should motivate the student to diligently apply these skills and abilities. Researchers have logically turned to a testable hypothesis (career versus game success) rather than attempting to demonstrate phenomenon that largely depend on perceptions of the individual. These most recent studies are encouraging, though results from the two studies cited were in conflict. Differing results may be related to the complexity of such studies. Of course, no simulation can tap all the characteristics or abilities required for a successful career. Moreover, certain factors may differentially affect the two success measures. For example, different motivations are likely in game play and career endeavors. Additionally, maturation of students between the time of game play and later career stages may affect the relationship between each success measure. Finally, many game plays reflect group efforts where variables associated with the group influence game success. Career success is often more individually based. In addition to these complications, relative career success measurement is complicated by the different criteria. Because of the lack of a consistent measure of subjective criteria such as approval or praise, it is difficult to precisely indicate significant variations in success when these criteria are employed. Using promotions or salary increases as indices of career success is considerably better; however, these criteria are also complicated by intercompany and intracompany transfers, organizational differences, industry factors, external economic conditions, confidentiality of information (an especially thorny issue in obtaining access to performance appraisals), the need to rely to a great extent upon self-reports with the attendant risk of bias, and difficulty in designing a valid research instrument. FUTURE RESEARCH INVOLVING EXTERNAL VALIDATION Clearly, additional study is needed relevant to the external validity of business games. Studies reported by Norris and Snyder [10] and Wolfe and Roberts [22] offer conflicting evidence regarding game success and career success. Further study is needed to clarify these findings. The limited research available suggests that academic skills as demonstrated by grade point average (GPA) and performance on achievement tests influence both game success and career success, and are therefore potential moderating variables. GPA appears to influence both game and career success, whereas performance on achievement tests is more strongly related to game success. Future studies of these relationships are warranted, with an awareness of the influence GPA and test performance may have on outcomes. Additionally, organization size appears to have an effect on career success measures. As might be expected, larger organizations tend to offer higher starting salaries and potentially greater promotional opportunities. To the extent that students equally successful at game play choose careers in organizations of varying size, this intervening variable may affect career success measures. Future studies might also place subjects in taxonomic categories on the basis of information received. Because of the problems inherent in conducting longitudinal investigations, it would be desirable to design a study that would control for as many intervening variables as possible. For example, it would be useful to segregate the subjects Developments in Business Simulation & Experiential Exercises, Volume 13, 1986 129 into various categories based on pronounced similarity of typical career patterns. if the business simulation is designed to hone skills of integration and overall system balance, a case might be made for stratifying those students who become entrepreneurs and use all of the integrative skills early in their careers. It also follows that future studies should have a large enough sample size to track students from a simulation situation to or through decidedly similar career paths. Thus, researchers would be able to compare performance or career success by category of career. Analysis of key success variables and correlation to key simulation success variables might then substantiate the transferability of simulated skills to the real- world career. A significant weakness in present research is the use of self- report measures of career success. Future attempts should obtain more objective measures of success, such as performance appraisals, supervisory ratings, and independent (archival) verification of salary increases and promotions, as well as other verification. Follow-up studies might include (1) specific questions on bonuses or other special incentive awards; (2) an assessment by respondents comparing their raise data and promotion data with that of others in their organization; (3) a categorization of respondents by functional area, such as technical, financial, or personnel; and (4) approximation or measure of the results of the respondents’ organizational unit, such as ROI or percent improvement. Further research is clearly warranted to incorporate these and other suggestions into future studies. REFERENCES [1] Babb, E. M., M. A. Leslie, and M. D. Slyke, The Potential of Business-Gaming Methods in Research,” Journal of Business, Vol. 39, 1966, pp. 465-472. [2] Boocock, S. 5, "Validity-Testing of an Intergenerational Relations Game,’ Simulation and Games, Vol. 3, 1972, pp. 29-40. [3] Bredemeier, M. E. and C. S. Greenblat, “The Educational Effectiveness of Simulation Games,” Simulation and Games, Vol. 12, 1981, pp. 307-332. [4] Byrne, E. T, “Who Benefits Most From Participation in Business Policy Simulations: An Empirical Study of Skill Development by Functional Area,” Proceedings of the Sixth National ABSEL Conference, New Orleans, Louisiana, 1979, pp. 257- 260. [5] Dukes, R. L., and S. J. Wailer, “Toward a General Evaluation Model for Simulation Games: GEM,” Simulation and Games, Vol. 7, 1976, pp. 75-96. [6] Greenlaw, P. 5., L. W. Herron, and R. H. Rawdon, Business Simulation in Industrial and University Education (Englewood Cliffs, New Jersey: Prentice- Hall, 1975). [7] Greenlaw, P. 5. and F. P. Wyman, “The Teaching Effectiveness of Games in Collegiate Business Courses,” Simulation and Games, Vol. 4, 1973, pp. 259-294. [8] Henshaw, R. C. Jr. and J. R. Jackson, The Executive Game (Homewood, Illinois: Irwin, 1972). [9] Hubbard, H. B., ‘Some Aspects of Management Gaming: A Panel Discussion,” in Simulation and Gaming: A Symposium, edited by Albert Newgarden. New York: American Management Association, 1961, pp. 38-59. [10] Kaufman, F. L., “An Empirical Study of the Usefulness of a Computer Based Game,” Journal of Education Data Processing, Vol. 13, 1976, pp. 13-22. [11] Kibbee, J. M., “Model Building for Management Games, in Simulation and Gaming: A Symposium, edited by Albert Newgarden. New York: American Management Association, 1961, pp. 8-15. [12] Kossach, C. F., “Some Aspects of Management Gaming: A Panel Discussion,” in Simulation and Gaming: A Symposium, edited by Albert Newgarden. New York: American Management Association, 1961, pp. 38-59. [13] Meier, R. C., W. T. Newell, and H. L. Pazer, Simulation in Business and Economics (Englewood Cliffs, New Jersey: Prentice- Hall, 19b9). [14] Mihram, G. A., Some Practical Aspects of the Verification and Validation of Simulation Models,” Operations Research Quarterly, Vol. 23, 1972, pp. 17- 29. [15] Mintzberg, H., The Nature of Managerial Work (New York: Harper & Row, 1973). [16] Norris, D. R. and C. A. Snyder, ‘External Validation of Simulation Games,” Simulation and Games, Vol. 13, 1982, pp. 73-85. [17] Parasuraman, A., “Assessing the Worth of Business Simulation Games, Simulation and Games, Vol. 12, 1981, pp. 189-200. [18] Pierfy, D., "Comparative Simulation Game Research: Stumbling Blocks and Stepping Stones, Simulation and Games, Vol. 8, 1977, pp. 255-268. [19] Vance, S. C. and C. F. Gray, “Use of a Performance Evaluation Model for Research in Business Gaming,” Academy of Management Journal, Vol. 10, 1967, pp. 27-37. [20] Wolfe, J., “Correlates and Measures of the External Validity of Computer-Based Business Policy Decision-Making Environments,” Simulation and Games, Vol. 7, 1976, pp. 411-438. [21] Wolfe, J., ‘The Teaching Effectiveness of Games in Collegiate Business Courses,’ Simulation and Games, Vol. 16, 1985, pp. 251-288. [22] Wolfe, J. and C. R. Roberts, “A Longitudinal Study of the External Validity of a Business Management Game,” Proceedings of the Tenth National ABSEL Conference, 1983, Tulsa, Oklahoma, pp. 9-12. Table of Contents Volume 13, 1986 How to Use Microcomputers in Human Resource Management Courses What's My Line? An Exercise in Job Analysis, Description and Classification Relationships Between Team Cohesion Dimensions and Business Game Performance Four Factors Affecting Group Performance in Business Policy Simulations Enhancing Strategic Management Goal-Setting Skills in the Business Policy Course Using a Business Simulation in the Principles of Management Course - Learning Outcomes and Perceptions Personal Computers: Drexel's Experience Using Psychological Type to Enhance Negotiation Skills The Effects of Conflict Handling Styles on Negotiation Behavior: An Analysis of an Experiential Exercise An Experiential Test of Bargaining Decisions and Representation Self-Grading of Examination by Students, A Viable Alternative to Traditional Procedures Perceived Instructor Enthusiasm and Student Achievement An Experiential Approach to Teaching Subordinate-Oriented Communication Assessing the Effects of a Computerized Study Guide in Macroeconomic Principles: A Statistical Analysis A Sales Management Simulation for the PC an Integrative Tool for the Sales Management Course Teaching Re-evaluation of Salespersons Through the Use of a Simulation Game A Test of Student Performance and Attitudes and Performance Under Varying Game Conditions Developing and Testing Airways: A Marketing Simulation Experiential Learning Revisited: Some Thoughts on Designing More Adaptive Management Education Programs Developing the Competencies of Creativity and Accurate Self-Assessment Self-View and Norms: Some Longitudinal Laboratory Results Strategic Planning with an Experiential Case The ABSEL Research Consortium: Preliminary Steps and Position Papers An Analysis of ABSEL Conference papers (1974-1985) The Creation and Operation of a Data-base System for ABSEL Research Funding for a Research Consortium on Simulation and Experiential Learning Simulation - Indoctrination or Learning The Dilemma In Evaluating Classroom Innovations Personality Variables on Group Cohesion, Team Participation and Total Learning Justifications for and Problems in Developing and Using Computerized Experiential Activities Management Competency Models and the Life-Long Learning Project: What Role for ABSEL External Validity of Business Games Restructuring the University - An Experiential Exercise The Surrogate Class Exercise Images of Effectiveness: A Classroom Exercise The Algebran Industry: A Tutorial Framework Another Chance: A Multi-Leveled, Macro Business Game for Micro-Computers A Water Quality Management Simulation Game Acquisitions? Divestitures? Progress Report on a Conglomerate Game Enterprise: A Multi-Purpose Management Development Simulation Marketer: A Microcomputer Simulation in a High Tech Industry A Pedagogical Analysis of Alternative Tax Elections for Exploration Costs in the Mining Industry Using of Multiple Microcomputers Application Programs to Teach Fundamental Business Concepts and Practices Demonstrating the Role of Simulation in Strategic Operations Planning: A Case Study in Bank Check Processing Location Analysis The Subjective Side of the Decision Support System: A Pitfall for the Panacea A Group-Based Procedure for Revealing Judgmental Heuristics and Biases An Experiential Exercise in Bayesian Decision-Making Values for Selected Parameters in Physical Distribution Simulation and Games Computerized Business Management Simulations for Tyros The Management Decision Laboratory at New York University An Example: The Use of Management Games on Microcomputers by Computer Novices Enhancing Mainframe Simulation Via Microcomputers: Designing Decision Support Systems Integrating Personal Computers Into a Course as a Decision Support Tool A Decision Support System for Capital Funds Forecasting Performance and Attitudinal Affects of a Decision Support Package in a Business Game Using the Real World: Interviews with Woman Mangers The Cooperatively-Authored Student Handbook International Buying: An Experiential Exercise The Research/Teaching Interface: Turning a Pretest into an Experiential Exercise Building Microcomputer Business Simulations Simulation with Discrete and Continuous Mathematical Modeling An Interpolation Approach to Developing mathematical Functions for Business Simulations The Design of Simulation Models Using Thought Organizers The Assessment Center as a Teaching/Learning Device An Approach to Meeting the AACSB Guidelines for Introducing Skills and Personal Characteristics (SACs) Using an Interclass Simulating and Role-Playing Pedagogy Experiential Learning about the World of Work: A Program for Primary and Secondary School Educators Assessment of Relevant Personal Characteristics of Potential Entrepreneurs