DETERMINANTS OF PERFORMANCE IN COMPUTER SIMULATIONS Developments in Business Simulation & Experiential Exercises, Volume 10, 1983 53 DETERMINANTS OF PERFORMANCE IN COMPUTER SIMULATIONS Jerry Gosenpud, University of Wisconsin-Whitewater Paul Meising, State University of New York at Albany ABSTRACT This study’s purpose was to identify factors which affect performance in computer-based business games. Its design entailed analyzing the relationship of 28 individual predictor variables to game performance. The predictor variables fell into six categories: academic ability, confidence, motivation, interest, cohesion and structure. Seventy two seniors playing The Tempomatic IV (Thompson and Strickland, 1980) participated. Eight predictor variables related significantly to performance: grade point average, major. whether teammates knew each other prior to the game, affection for teammates, degree to which teammates worked together as opposed to autonomously, degree to which team decision making was formal, degree to which teams were well organized, desire to play the game as it progressed. INTRODUCTION As in any academic experience, some students perform better than others in computer based business games; and, as in most academic experiences, why this is true has not been completely substantiated. The purpose of this study was to discover why some students perform better than others in such simulations. This is important for a variety of reasons. First, participation in a business game is often a new experience in which many students do not do well. If the factors affecting performance could be isolated, it may be possible to help those who have been identified as potentially low performers. Second, since business games are typically played in teams, certain team-related characteristics probably facilitate success. For example it could be that teams are more successful if strong leadership exists. If such characteristics can he identified by research, then instructors could help teams improve by encouraging them to adopt these success-facilitating characteristics. Third, computerized games are designed to simulate real business conditions. Assuming that they do, research results suggesting who performs well and who does not in these business games may be generalizeable enough to predict success in other types of business situations. The belief by many that success in business games is due almost entirely to luck provides a fourth reason for this study, Research should investigate this notion--research designed to explore and cut through the capriciousness associated with business games and separate the factors systematically related to game performance from those which are random. Possible Factors Individual and team characteristics are two possible explanations for why some students do better than others in computer simulations. Individual characteristics are important because the simulation is an academic experience. Hence, it is possible that the factors which may affect success in it are the same individual characteristics which influence success in other academic experiences, i.e., academic ability, motivation, interest and confidence. In other words it may be that those with the greatest interest, motivation, confidence and ability to do well will indeed perform the best. The second set, team characteristics, is important because business games are often played in teams. Hence, it is possible that certain team characteristics will affect success. Group cohesion and group organization are two such attributes which have been identified in the literature. Some authors suggest that cohesive teams perform better than others, while others suggest that structured teams perform better. Previous Research There is little previous evidence regarding the impact of motivation, interest or confidence on game performance. However, there is evidence available that game performance is associated with academic achievement as measured by overall CPA (Seginer, 1980; Vance end Gray, 1967; Wolfe, 1978) and with aptitude as measured by an ETS admissions test (McKenney and Dill, 1966). Seginer (1980) however argues against that conclusion. She believes the relationship between general academic performance and game performance is complex and that the abilities which underlie game performance may be different from those underlying general academic performance, The evidence regarding the effects of cohesion and organization is inconclusive. On one hand, Wolfe’s (1975) examination of the behaviors associated with success in games indicated that both cohesion facilitating behaviors (such as cooperativeness, sharing of information and free discussion) and structure facilitating behavior (such as the creation of clear policies and businesslike meetings) were associated with success. Miesing’s research (1982) also indicates that success is greatest when both cohesion and structure are high. However, both are necessary for success according to his results; when one or the other is absent, groups are lees successful. On the other hand, other studies report negative results. Brand (1980) found no relationship between cohesion and game related performance, and Hutte (1964) found no relationship between game success and structured organization. THE PRESENT STUDY AND ITS VARIABLES The present study was an effort to ascertain factors which affect computerized business game performance. In it, six variables were identified which may be predictive of performance in the business game. They were Academic Ability, Confidence, Motivation, Interests, Cohesion and Organization. In this study, all variables were measured by one of two methods: (1) a questionnaire consisting of Likert- type, forced choice and essay questions; or (2) content analysis of a semester journal kept by the students. Predictor Variables Academic Ability - College GPA. Confidence - The students’ expressed confidence to do Developments in Business Simulation & Experiential Exercises, Volume 10, 1983 54 well in the game as reported on a Likert-type questionnaire item. Motivation - The students’ goals for the course, interest level in playing the game, and desire to do well. Motivation was measured in the following ways: (a) a Likert question on desired course grade, (b) an essay question on continuing desire to play the game, (c) an essay question covering course goals, (d) and a forced choice question dealing with student reasons for choosing their teammates. (If the student chose teammates as the basis of high expected performance, it was presumed indicative of a high desire to do well, but if the student chose teammates on the basis of friendship or proximity, it was presumed that needs other than achievement were being fulfilled.) Interests - The students’ academic or professional interests and situational factors which might affect their interest in the course or game. Specific indices included major, date of graduation, whether students intended to work or go to graduate school after graduation, and whether or not they had already attained a job or graduate school admittance (all measured by forced choice questionnaire items). Cohesion - This class of variable tapped how well teammates knew each other before the class and the cohesiveness of teams during the game. Cohesiveness is a general variable which included individual Likert questions measuring the friendliness and the harmony of team relations, the openness and trustingness of the team’s atmosphere and the degree to which decision discussions were thorough. This general variable also included how often group members chose each other’s company as measured by an essay question, whether there was conflict and how it was handled as expressed in the students’ journal and how much the student liked other group members as indicated in the journal. Structure - Group structure included specific variables indicating the degree to which tasks were clearly allocated, decision-making was formalized, activities were clearly directed, strong leadership was established and the group was organized and methodological. All variables were measured with Likert questions. Dependent Variables The study’s outcome variable in this study was the student’s numerical grade in the computerized business simulation. However, since the relationship of any of the predictor variables to other course outcomes may shed light on the impact of that predictor variable, other course outcomes were also measured. These included satisfaction with the group, satisfaction with performance, and final course grade. Grades were obtained from instructors, and satisfaction indices were measured by Likert responses. METHOD Subjects The subjects consisted of 72 seniors from 2 sections of Business Policy at the University of Wisconsin- Whitewater. These students played the Tempomatic IV Simulation (Scott and Strickland, 1980), and game performance was worth 25% of the course grade. Procedure On the second day of the term, students were Introduced to the business game and the research project, asked to choose themselves into teams of three, told of the researcher’s general purposes, arid asked to fill out an initial questionnaire. The initial questionnaire requested data indicating major, GPA, plans after graduation, course goals, confidence, and reasons for choosing teammates. During the second week of the game, they were asked to keep a journal of events related to game participation both in and outside of their groups. Students played the game in teams for five weeks, and continued as single member firms. As indicated in Figure 1, market curves for each of the new industries continued from the curve of the old industry, but new industry curves differed from each other. After the team members separated, each player was asked to fill out a “follow up” questionnaire regarding group cohesion, organization, leadership and satisfaction. At the end of the game students were asked to turn in their journals and fill out a final questionnaire. This questionnaire asked students if they worked with anyone since the teams separated and whether their effort and interest in the game changed from the onset. Grades were collected at the end of the term. Analysis Three types of statistics are used to evaluate the results. Discrete predictor variables containing only two categories were subjected to t-tests. Discrete predictor variables containing more than two categories were subject to analyses of variance. These analyses are depicted by F-tests and appear in Table 1. For continuous predictor variables relationships are depicted by correlation coefficients and appear in Table 2. T-test results are presented in the text of this report and do not appear in tabular form. TABLE 1 ANALYSIS OF VARIANCE OF GAME PERFORMANCE BY MULTIPLE CATEGORY DISCRETE PREDICTOR VARIABLES df F P MOTIVATION Course Goals 71 0.66 .679 INTEREST Major 71 2.43 .029 Expected Grad. Date 70 1.82 .169 COHESION Conflict Level 46 0.88 .517 Teammate Affection 48 2.34 .052 Developments in Business Simulation & Experiential Exercises, Volume 10, 1983 55 RESULTS Previous Academic Performance. As indicated in Table 2, game performance correlated significantly with previous CPA. The final grade in the course also varied positively and significantly (r=.31, p<.00l) with previous GPA. This result suggests that performance does vary with previous academic performance. Confidence. Table 2 reveals no correlation between game performance and confidence. This suggests the lack of a generalizeable relationship between game performance and confidence. Motivation. Table 2 shows a strong positive correlation (r. 51, p <.001) between game performance and expressed desire to play the game as it progressed. It is not conclusive from this result that increased desire simply leads to better performance. This is because desire to play was measured at the end of the game and not continuously throughout it. Thus the variable, desire, revealed students’ reflections as the game was concluding and these reflections could have been influenced by performance. The result indicating a high correlation between performance and desire, then, probably suggests a mutually reinforcing relationship between desire and performance. More often than not high desire leads to successful performance which further heightens desire. High desire can lead to low performance, though, and when it does, the result may be reduced desire and even lower performance. Came performance did not vary with any of the other measures of motivation. As shown in Table 2, correlations between performance in the game, and the variables measuring the degree to which students were motivated to do well in the game (i.e., grade desired and degree to which teammates were chosen for performance- related reasons) were near zero. Also, Table 1 shows that game performance did not vary with students’ goals for the course. Interest. As indicated in Table 1, the F-test shows that game performance varied significantly with major. Least- significant-difference contrast tests show that accounting majors scored significantly higher (p.05) than other majors including marketing and management and general business majors. This result suggests that students with extensive experience with financial statements perform better than students without such experience. Game performance did not vary with other variables in the interest category. The F-test in Table 1 found no relationship between game performance and expected graduation date. Also, t-tests found no relationship between game performance and either expected post- graduation activity (t- 0.09, p-.935) or whether students had already attained employment or acceptance to graduate school (t=0.66, p=.627). However, another of this study’s outcome variables, final course grade, did vary with whether students had an assured job or attained graduate school acceptance. Those with assured jobs or graduate school acceptance attained significantly higher course grades than those without them (t=2.44, p=.018). This relationship probably indicates a similarity between attributes necessary for policy course performance and attributes sought by graduate schools and prospective employers. Cohesion. Table 1 shows that game performance varied significantly with level of affection as indicated in student journals, and a posteriori contrast test (least-significant- difference tests) show that performance was significantly higher (p. 05) for those expressing an affinity with their teammates than those who expressed “dislike.” Also, as indicated in Table 2, game performance varied significantly (r-.25) with the degree to which team members worked together as opposed to autonomously and with how well teammates knew each other before class (r=.28). On the other hand, as indicated in Table 2, game performance did not vary with the responses to nine of the Likert items measuring cohesion and a composite index of cohesiveness. Also, performance was no better for those who met with their teammates after the teams disbanded (an indication of team cohesion) than for those who did not meet (t=1.11, p.28). As one might expect, cohesion indices varied strongly with satisfaction indices. Of 22 correlations between cohesion items and satisfaction items, all were significant at the .05 level. Eighteen of these correlations were greater than .40, and nine greater than .60. Structure. Table 2 shows significant correlations between game performance and the degree to which the group was perceived to have formal decision making (r’ .25) and good organization (r.23). Came performance did not correlate significantly with other group organization measures. Of the seven specific indices of organization, five correlated at the .06 level with performance satisfaction. Correlations ranged between .19 and .41 and show a clear student preference for well-organized groups. The sixth index of organization, defined leadership, correlated negatively with team member satisfaction (r=-. 34, p,003) and near zero with performance satisfaction. Apparently the preference for structured groups does not include the desire for one dominant leader. Developments in Business Simulation & Experiential Exercises, Volume 10, 1983 56 DISCUSSION The intent of this study was to identify those variables which would help predict which students would perform well and poorly in a computerized business simulation. The study’s design entailed analyzing the relationship of 28 individual predictor variables to game performance. The results show that eight of the predictor variables related to game performance: 1. GPA, 2. Major, 3. Whether teammates knew each other prior to the course, 4. Affection for teammates, 5. Degree to which team members worked together as opposed to autonomously, 6. Degree to which decision making was formal, 7. Degree to which the group was well organized, and 8. Desire to play the game as it progressed. Given these results, it appears that students will perform well in a computerized business game if they enter with high GPA’s, are accounting majors, and team up with people they know prior to class. The results also suggest that teams will perform well if they are well-organized, engage in formal decision making, and teammates like each other and work together. Finally, the results suggest a strong relationship between game performance and desire to play the game as it progresses. One intention of this study was to further investigate the notion that success in business games is due entirely to luck by using research to identify tangible factors or independent variables which may affect computer game performance. As indicated above, eight such factors were found to vary significantly with performance. But the question remains whether enough of the variance in game performance can be explained by its linear dependence upon the eight independent variables operating jointly. To answer, a multiple correlation coefficient was computed to determine the degree of dependence of game performance on the eight independent variables. This coefficient (Multiple R) was .697 indicating that 48.6% of the variance associated with game performance can be explained by the eight combined independent variables, suggesting strongly that game performance is due to factors other than luck. Another intention was to apply this study’s results to the classroom to help those who might perform poorly in the game. According to the results, those moat likely to perform poorly and need help: (1) had lower GPA’s before entering class; (2) had majors not requiring extensive experience with financial statements; and (3) were on teams with strangers. Still another of this study’s intentions was to identify team characteristics associated with success. In this study the most successful teams had: (1) good organization; (2) formalized decision making; and (3) good task relationships. Applied to the classroom, this means that instructors can encourage teams to develop the above characteristics. There are methodological limitations to this study which future research should seek to rectify. First its sample was 72 students from one university, and the generalizability of the results is therefore suspect. This is especially true given that different universities use different simulations and assign different weights to performance in the game. This study’s results need to be replicated at different universities with different games in order to be generalizeable. Second, as indicated above, students were asked about their desire to play the game after it was over, and their answer to that question may have been influenced by their performance. To reduce such influence, questions regarding desire to play the game in future studies should be asked nearer to the game’s beginning. Finally, data collected regarding cohesion and structure came from teams who were only together for five weeks of a twelve week game. Perhaps research examining groups together for longer periods of time would produce stronger correlations between performance and team cohesion and structure. REFERENCES (1) Brand, Charles F. “Learning from Simulation Games: Effects of Sociometric Groupings,” Simulation and Games, 1980 (June), 11 (2), 163-176. (2) Hutte, Herman. “Decision-making in a Management Game,” Human Relations, Vol. 18, No. 1, 1964, pp. 5- 20. (3) McKenney, James L. and Dill, W. R. “Influence on Learning in Simulation Games,” The American Behavioral Scientist, 1966, 10 (2), 28-32. (4) Miesing, Paul. Qualitative Determinants of Team Performance in a Simulation Game, Proceedings of the Ninth Annual Meeting of the Association for Business Simulation and Experiential Learning, 1982, pp. 228- 231. (5) Scott, Charles R., and Strickland, Alonzo J. Tempomatic IV: A Management Simulation, (Boston: Houghton Mifflin, 1980, 2nd edition). (6) Seginer, Rachel. “Game Ability and Academic Ability: Dependence on SES and Psychological Mediators,” Simulation and Games, 1980 (Dec.), 11 (4), pp. 403- 421. (7) Vance, S. C., and Gray, C. F. “Use of Performance Evaluation Model for Research in Business Gaming,” Academy of Management Journal, 1967, 10, pp. 27- 37. (8) Wolfe, Joseph. “Correlations Between Academic Achievement, Aptitude and Business Game Performance,” Proceedings of the Sixth Annual Meeting of the Association of Business Simulation and Experiential Learning, 1978, 316-324. (9) Wolfe, Joseph. “Effective Performance Behaviors in a Simulated Policy and Decision Making Environment,” Management Science, Vol. 21, No. 8, 1975, pp. 872- 882. Table of Contents Volume 10, 1983 Is the Computerized Business Simulation Relevant? Business Professionals Play a Student Game A Methodology For Assessing the Internal Validity of Business Simulations A Longitudinal Study of the External Validity of a Business Management Game Policy Analysis and Decision: A corporate Relocation Simulation Exercise Effective Listening: An Exercise in Managerial Communication The Symbol Exercise: An Initial Group Activity Concept Based Simulations Institutional Users of Experiential Learning Packages: A Preliminary View from Publishers' Adoption Lists How to Use Business Games in the Business Policy Course: The Students' Perspective Professors' Ratings of Business Policy Learning Methods Moot Trial: An Exercise in Trial Procedure and Evidence The Advertising Agency Game: An Experiential learning Exercise The Use of Videotaped Cases in Teaching Information Acquisition and Decision-Making Skills Experiencing Information Processing Strategies as a Means to Explore Decision Making Importance Ratings and Operations Data as Predictors of Business Game Performance Determinants of Performance in Computer Simulations Predicting Business Game Performance form Perceptions of Manager Information and actions Business Consulting: A Practicum for Undergraduate Internship as a Contingency Based Experiential Learning Program for More Effective Organizational Socialization: A Conceptual Framework MANSYM III Decision Support System Demonstration Learning the Concept of market Value Through Simulation Conflict Management for Economic Developers Development of Data Analysis Units Designed to Enhance Reasoning and Knowledge Transfer in the College Level Course BOSS: A Behavioral-Quantitative, Computer-Supported Game Entrepreneurial Potential: An Experiential Exercise in Self Analysis and Group Assessment Development of Strategists: Simulated Cases BANKRUPT: A Deceptively Simple Business Strategy Game Simulating Market and Firm Level Demand - A Robust Demand System COMPSIM A Computer Center Management Simulation Hiving Model: Assessing Management Skill Awareness The Johari Window, A Reconceptualization Role-Playing Based on Video-Tape Scenarios: An Application of Modeling to Building Supervisory Skills How to Internationalize Your Curriculum A Computerized Model of Human Behavior in a Total-Firm Management Simulation the Worksheet Approach for Simulation Game Strategy Analysis Teaching Competitive Bidding Using a DSS Generator Do We Learn from Experience? The Use of Theory Power for Increased Research Momentum in Business Simulation and Experiential Exercises Research Report on Programmatic Research on Perceived Learning Barriers with Simulation and Experiential learning An Empirical Examination of Conflict - and Nonconflict - Oriented Problem-Solving Technologies Management Curriculum: 1982 Humor as a Management Tool: Use in Formal Game Presentations Student Behavioral Change Through Teacher behavioral Change