A LONGITUDINAL STUDY OF THE EXTERNAL VALIDITY OF A BUSINESS MANAGEMENT GAME Developments in Business Simulation & Experiential Learning, Volume 10, 1983 9 A LONGITUDINAL STUDY OF THE EXTERNAL VALIDITY OF A BUSINESS MANAGEMENT GAME Joseph Wolfe, University of Tulsa C. Richard Roberts, University of Tulsa ABSTRACT A five-year follow-up study investigated relationships between student performance in a top-management game and various measures of business career success. Positive results were found between numerous predictor variables, Relationships between course-related features and other predictors of career success were also examined. A number of fairly rigorous studies on the internal validity of business games have been performed since their advent over two decades ago. Although mentioned only briefly in one of gaming’s early conferences [6, p. 67], their external validity has been directly questioned in recent years [14; 15; 18]. Occasional research has addressed this topic but the designs used invariably classify them as quasi-external validity studies. One method compares the behaviors of students with those of successful business executives playing the same simulation [2; 3; 19]. A second method contrasts the traits of successful student players with those of successful executives [16] while a third method ex- wines the quality range of play obtained by executives who have been differentially successful in their business careers [4; 12; 16]. In total these studies have provided circumstantial evidence supporting the external validity of a business game experience. Although these results are encouraging for game users, these studies fail to directly test the relationship between a student’s gaming performance and that same student’s later business career performance. Longitudinal studies of this type have often been called for but only recently have Norris and Snyder [13] attempted this difficult ideal. Data obtained from a five-year follow-up questionnaire determined that there were no correlations between their game performance variable, an index composed of the firm’s ending return on investment and general game participation, and student career success as measured by (1) number of promotions received, (2) levels removed that student was from the chief executive officer, and (3) percent salary change since graduation. Although their study is an important effort, certain methodological problems could have caused the lack of correlation found. On the predictive side of their study it has been subsequently determined that the simulation they employed, Henshaw and Jackson’s The Executive Game [10], was relatively barren in a study of the effects of three levels of game complexity on knowledge acquisition [21]. Secondly, individual player performance was not measured as participants were graded by teams, and lastly, the rate-of- return on investment (Rot) dominated the grading criterion and other studies have found different standards to be more indicative of game performance quality [9; 20]. On the career success side of their study certain measures were quite appropriate but a student’s distance from the firm’s chief executive officer is contaminated by organizational size effects. While Norris and Snyder recognized the difficulties of measuring a construct as ambiguous as “career success,” career satisfaction and absolute salary level changes are accessible and important indicators of success. This paper presents an investigation into the long-term relationship between student game performance and four alternative measures of business career success. Additional indicators of college achievement are also presented to better understand the causal chain being derived. METHODOLOGY The subjects were 142 graduating business seniors playing Jensen and Cherrington’s The Business Management Laboratory [11] for ten weekly decision rounds in a capstone business policy course. This simulation was chosen as it bad been found to be motivating, comprehensive, and functionally unbiased in other research applications. All subjects had agreed to participate for five years--a period considered adequate to reduce the early halo effects of grade-point averages, career switches, graduate school attendance, and marriages on ultimate salary levels and career advancement, while still tapping the possible long-term career effects of skills and concepts acquired and demonstrated in a business game. Demographic information was collected in class, aptitude test scores (ACT) and grade- point averages (GPA’s) were obtained from admissions records, and salary expectations and earnings histories were compiled through the use of telephone interviews and self- reported mailed questionnaires. The study controlled for class size, textbook, case assignments, lecture topics, grade weights, meeting tines, industry size, and the instructor. Table 1 displays the response rate ultimately obtained from two mailings. Appropriate Kruskall-Wallis one-way ANOVA and median- tests were conducted to determine if respondents were significantly different from those who did not respond or could not be located through wrong addresses and non- forwardable questionnaires. No differences by any characteristic such as age, sex, college major, salary level, game performance, career satisfaction, course value, number of promotions, grade-point-averages, or ACT scores were found. It was concluded that bad addresses were randomly distributed and that a non-response TABLE 1 RESPONSE CATEGORIES Study Group Number Percent Population 142 100.0 Bad address 35 24.6 Non-respondent 25 17.6 Respondent 82 57.8 Developments in Business Simulation & Experiential Learning, Volume 10, 1983 10 bias did not exist. Approximately 24.6% of the original student population could not be traced while the true response rate was 82.4%. Various career success measures were collected to capture the concept’s multidimensionality. The criteria attempted to gauge economic rewards obtained, upward within- organization mobility, relative monetary gain, and subjectively appraised job, salary, and responsibility as suggested by Hall [8], Astin [1], Wolfe [22], Campbell [5], Gurin, Veroff and Feld [7], and Williams and Harrell [17] among others. 1. Salary-- the student’s absolute salary five years after graduation as a measure of the individual’s economic value. 2. Percent salary change-- the respondent’s percent increase in salary since graduation as a measure of relative economic improvement. 3. Promotions-- the number of promotions the respondent received as a measure of upward mobility within an organizational setting. 4. Satisfaction-- expressed job satisfaction as measured by the respondent’s self-assessed degree of underemployment in salary level, job responsibility, and the job Itself. Game performance was measured by three global and inter- related economic criteria-- total earnings, and rates-of-return on investment (ROI) and owner’s equity (ROE). Participants were assigned to single-member firms in 8-firm industries. First-place firms were ranked 1 while last-place firms were ranked 8 according to their respective economic results. A performance index comprising all three measures is also reported here as it represented 55% of the course’s overall grade and was the student’s grade-optimization goal. RESULTS Table 2 presents Pearson product moment correlations between game performance and the various measures of career success. All game performance indicators were positively related to salary levels and job satisfaction; the results for salary changes and promotions were in the correct direction and approached significance. Table 3 presents correlations between aptitude (ACT scores) and academic achievement and career success. These measures were taken as they represent often-used predictor variables in career and college- related success research. ACT’s were significantly related to the degree of five-year salary change but in the opposite direction expected; the signs were also in the unexpected direction for promotions. Student academic achievement by major subject area and total business school courses was positively associated with salary levels and salary change. Table 4 and 5 show the correlations found between aptitude, academic achievement, and game performance-- measures often used in internal game validity studies. Significant and positive relationships were found between 13 of 16 aptitude/achievement associations while all correlations between game performance and academic achievement were significant in the expected direction. DISCUSSION Although the results presented here strongly support the particular teaching application’s external validity, TABLE 2 GAME PERFORMANCE 2 Game Criteria Salary % Change Promotions Satisfaction Earnings -. 249b -.207 -.197 - .176a ROI -. 238b -.157 - .107 -. 175a ROE -.276c -.145 -.104 198a Index -. 267c - .193 - .162 191a 1The following significance levels apply to all tables: asignificant p .05 bsignificant p<.01 cSignificant p <.001 dSignificant p <.0001 2Firms ranked by placement within industry; First- place 1, last-place - 8. TABLE 3 APTITUDE AND COLLEGE PERFORMANCE ACT Salary % Change Promotions Satisfaction English -.027 - 395b - .251 - .007 Math .139 320a -.091 .067 Soc. Sci. .159 -.155 -.153 .092 Composite .137 -. 283a -.142 .093 GPA Salary % Change Promotions Satisfaction Major .254b 252a .020 .143 Q. Meth. .061 .109 -.176 -.023 Bus. Sch. .187a .138 -.093 .082 University .154 .130 -.075 .081 TABLE 4 INTERCORRELATIONS OF APTITUDE AND COLLEGE PERFORMANCE GPA ACT Major Q. Meth. Bus. Sch. University English 240a .172 •316b 387d Math 226a 255a 267a .220 Soc. Sci. 250a .209 287b 324b Composite 327a .271 352c 387c the game’s relative career impact has not been deter- minded. Step-wise multiple regression was employed to determine significant relationships between sets of predictor variables and individual career performance variables. The marginal variations in performance explained by each predictor variable were systematically examined to determine the relative significance of each regression coefficient (alpha = .05). Intercorrelations between variables were examined at each step to reduce potential multicollinearity (tolerance - .6). Developments in Business Simulation & Experiential Learning, Volume 10, 1983 11 TABLE 5 INTERCORRELATIONS OF APTITUDE AND GAME PERFORMANCE GPA Game Criteria Major Q. Meth. Bus. Sch. University Earnings -.435d -.320 c -.459 d -.450 d ROI - 397 d - 360 d - 447 d - 418 d ROE 379 d -.320 c - 404 d -.367 d Composite -.440 d -.347 d -.471 d -.449 d Table 6 presents the results of this process by success measure. In each case game earnings made a significant explanatory contribution although a student’s basic English aptitude score and/or major-area grade- point achievement carried more weight. The game's long-term impact, however, was quite remarkable given that it occupied about one-half of one course’s teaching inputs out of over 40 baccalaureate courses taken for the business degree. Although a modest degree of career success variance was explained, some types of success were more completely predicted. Approximately 92.2% of an individual’s degree of job satisfaction was not explained and at best only 38.2% of the students’ salary level was accounted for. The role of industry- and company- choice, motivation, luck, and developmental experiences, among other factors must be considered for a more complete description of the path to career success. Overall though, it appears that college is indeed what Astin [1] called “four critical years’ and that a business game adds both short- and long-term meaning to those years. SUMMARY A fairly large-sampled external validity study found that successful business game play was associated with successful business careers when measured in terms of salary levels and job satisfaction. The business game seemed to implement those skills and cognitions which had previously led to academic achievement. These abilities in turn were carried into real-world careers. Game earnings were a significant predictor when forecasting career success although low English aptitude and high academic achievement in the student’s major subject area were even more important predictors. REFERENCES [1] Astin, Alexander W., Four Critical Years, (San Francisco: Jossey-Bass, 1978). [2] Babb Emerson N., and Ludwig M. Eisgruber, Purdue Dairy Management Game, (Chicago: Educational Methods, 1966). [3] Babb, Emerson M., and Ludwig N. Eisgruber, Purdue Farm Supply Business Management Game, (Chicago: Educational Methods, 1966). [4] Babb, Emerson M., M.A. Leslie, and M.D. Van Slyke, The Potential of Business-Gaming Methods in Research,’ Journal of Business, Vol. 39, No. 4, 1966, pp. 465-472. TABLE 6 SIGNIFICANT PREDICTOR VARIABLES Success Criteria Predictor Variables Explained Variance Salary 1. GPA Major 2. ACT English 3. Game earnings 38.2% Salary Change 1. ACT English 2. GPA Major 3. Game program 26.9 Promotions 1. ACT English 2. GPA Major 3. Course lectures 4. Game earnings 28.1 Satisfaction 1. ACT Composite 2. GPA University 3. Game earnings. 4. Course lectures 7.7 [5] Campbell, Donald P., A Study of College Freshmen-- Twenty-Five Years Later, (Minneapolis: Cooperative Research Program, University of Minnesota, 1965) [6] Dill, William R., James R. Jackson, and James W. Sweeney (eds.), Proceedings of the Conference on Business Games as Teaching Devices, (New Orleans: Tulane University, 1961) [7] Gurin, Gerald, Joseph Veroff, and Shiela Feld, Americans View Their Mental Health, (N.Y.: Basic Books, 1960). [8] Hall, Douglas T. , Careers in Organizations, (Pacific Palisades, Calif.: Goodyear, 1976). [9] Hand, Herbert H., and Henry P. Sims, ‘Statistical Evaluation of Complex Gaming Performance,” Management Science, Vol. 21, No. 6, 1975, pp. 708- 717. [10] Henshaw, Richard C., and James R. Jackson, The Executive Game, (Homewood, Ill.: Irwin, 1972). [11] Jensen, Ronald L., and David J. Cherrington, Business Management Laboratory, (Dallas: Business Publications, 1973) [12] McKenney, James L., and William R. Dill, ‘Influences on Learning in Simulation Games,’ American Behavioral Scientist, Vol. 10, No. 3, 1966, pp. 28-3 2. [13] Norris, Dwight R., and Charles A. Snyder, “External Validation of Simulation Games," Simulation & Games, Vol. 13, No. k, 1982, pp. 73-85. [14] Parasuraman, A., Evaluation of Simulation Games: A Critical Look at Past Efforts and Future Needs, In Daniel C. Brenenstuhl and William D. Biggs (eds.), Experiential Learning Enters the 80's (Tempe: Arizona State University, 1980), pp. 192-194. [15] Snow, Charles C., “A Comment on Business Policy Teaching Research,” Academy of Management Review, Vol. 1, No. 4, 1976, pp. 133-135. [16] Vance, Stanley C., and Clifford F. Gray, ‘Use of Developments in Business Simulation & Experiential Learning, Volume 10, 1983 12 a Performance Evaluation Model for Research in Business Gaming," Academy of Management Journal, Vol. 10. No. 3, 1967, pp. 27-37. [17] Williams, Frank J., and Thomas W. Harrell, Predicting Success in Business,” Journal of Applied Psychology, Vol. 48, No. 3, 1964, pp. 164-176. [18] Wolfe, Joseph, “A Response to Snow's Comment on Business Policy Teaching Research,” Academy of Management Review, Vol. 1, No. 4, 1976, pp. 136- 137. [19] Wolfe, Joseph, “Correlates and Measures of the External Validity of Computer-Based Business Policy Decision Making Environments,” Simulation & Games, Vol. 7, No. 4, 1976, pp. 411-438. [20] Wolfe, Joseph, “Correlations Between Academic Achievement, Aptitude, and Business Game Performance,’ in Daniel C. Brenenstuhl and Samuel C. Certo (eds.), Exploring Experiential Learning: Simulations and Experiential Exercises, (Tempe: Arizona State University, 1978), pp. 316-324. [21] Wolfe, Joseph, “The Effects of Game Complexity on the Acquisition of Business Policy Knowledge,” Decision Sciences, Vol. 9, No. 1, 1978, pp. 143-155. [22] Wolfe, Dael, “To What Extent Do Monetary Returns to Education Vary with Family Background, Mental Ability, and School Quality?” in Lewis C. Solomon and Paul J. Taubman (eds.), Does College Matter? (N.Y.: Academic Press, 1973). Table of Contents Volume 10, 1983 Is the Computerized Business Simulation Relevant? Business Professionals Play a Student Game A Methodology For Assessing the Internal Validity of Business Simulations A Longitudinal Study of the External Validity of a Business Management Game Policy Analysis and Decision: A corporate Relocation Simulation Exercise Effective Listening: An Exercise in Managerial Communication The Symbol Exercise: An Initial Group Activity Concept Based Simulations Institutional Users of Experiential Learning Packages: A Preliminary View from Publishers' Adoption Lists How to Use Business Games in the Business Policy Course: The Students' Perspective Professors' Ratings of Business Policy Learning Methods Moot Trial: An Exercise in Trial Procedure and Evidence The Advertising Agency Game: An Experiential learning Exercise The Use of Videotaped Cases in Teaching Information Acquisition and Decision-Making Skills Experiencing Information Processing Strategies as a Means to Explore Decision Making Importance Ratings and Operations Data as Predictors of Business Game Performance Determinants of Performance in Computer Simulations Predicting Business Game Performance form Perceptions of Manager Information and actions Business Consulting: A Practicum for Undergraduate Internship as a Contingency Based Experiential Learning Program for More Effective Organizational Socialization: A Conceptual Framework MANSYM III Decision Support System Demonstration Learning the Concept of market Value Through Simulation Conflict Management for Economic Developers Development of Data Analysis Units Designed to Enhance Reasoning and Knowledge Transfer in the College Level Course BOSS: A Behavioral-Quantitative, Computer-Supported Game Entrepreneurial Potential: An Experiential Exercise in Self Analysis and Group Assessment Development of Strategists: Simulated Cases BANKRUPT: A Deceptively Simple Business Strategy Game Simulating Market and Firm Level Demand - A Robust Demand System COMPSIM A Computer Center Management Simulation Hiving Model: Assessing Management Skill Awareness The Johari Window, A Reconceptualization Role-Playing Based on Video-Tape Scenarios: An Application of Modeling to Building Supervisory Skills How to Internationalize Your Curriculum A Computerized Model of Human Behavior in a Total-Firm Management Simulation the Worksheet Approach for Simulation Game Strategy Analysis Teaching Competitive Bidding Using a DSS Generator Do We Learn from Experience? The Use of Theory Power for Increased Research Momentum in Business Simulation and Experiential Exercises Research Report on Programmatic Research on Perceived Learning Barriers with Simulation and Experiential learning An Empirical Examination of Conflict - and Nonconflict - Oriented Problem-Solving Technologies Management Curriculum: 1982 Humor as a Management Tool: Use in Formal Game Presentations Student Behavioral Change Through Teacher behavioral Change