CORRELATES OF LEARNING IN SIMULATIONS Developments In Business Simulation & Experiential Exercises, Volume 23, 1996 CORRELATES OF LEARNING IN SIMULATIONS Jerry Gosenpud and John Washbush University of Wisconsin-Whitewater ABSTRACT This study attempted to identify variables that correlate with learning in simulations. The researchers explored whether simulation learning varied with (1) simulation performance, (2) the degree to which players were struggling with the simulation, (3) type of simulation goals, and (4) common sense variables often associated with learning such as confidence. The subjects were college seniors; the simulation lasted eleven quarters; learning was measured by Instructor designed instruments; other variables were measured by questionnaire. The results were that students who expressed game related financial goals, such as to maximize profits, early in game learned more and that those that perceived the game to be understandable and simple early in the game also learned more. Learning did not correlate with performance and the degree of struggle. INTRODUCTION This is the fourth in a series of studies exploring the correlates of learning in Total Enterprise (TE) Simulations. All studies took place in the college classroom and are generalizable only in those settings. There are two general purposes for this research. The first is to understand why some students learn more than others In TE simulations or, to state in another way, identify behaviors and variables that are associated with greater learning in the simulation environment. The second is to determine whether there is a relationship between performance in the simulation, as measured by profit-related variables, and learning. Previous Literature Simulations are learning tools and are used extensively in learning environments, particularly in colleges and for training. One would expect then that learning would be the focus for a great number of research studies, and for business simulations that has been the case. Unfortunately, most of the research previously undertaken has not been helpful for the purposes of the present research. Learning was a key dependent variable in some of the early research establishing the validity of simulations. Such researchers as Brenenstuhl and Catalanello (1979), Burns and Sherrell (1984), and Wolfe (1976) compared test scores of cognitive learning from lecture, case, and simulation sections of the same course. While such studies used learning as the criterion for such comparisons, their purpose was to assess the teaching methodologies and not to understand how simulations were helping students learn. Studies focusing on factors influencing learning in simulations began to appear in 1989. Whiteley and Faria (1989) and Faria and Whiteley (1990) found that simulation games are effective in improving quantitative skills but not so in improving the acquisition of applied knowledge. Wellington and Faria (1991) examined the relationship between simulation participation, level of performance in a simulation competition, and recency of play with exam scores (presumably a learning measure). They found no relationship between simulation play and exam scores, level of simulation performance and exam scores, and recency of simulation play and exam scores. They suggested that simulation play involves skills which may not be directly measurable by normal multiple choice exams. Carvalho (1991) statistically examined the possibility of developing a learning validation model, which would permit instructors using simulations to develop course objectives with greater clarity by employing known characteristics of chosen games. The above studies appear to concentrate on the value of the simulation as a learning tool and on the kinds of learning enhanced with simulations. In contrast, this study presumes that learning takes place in simulations. It asks what factors enhance greater learning. Learning and Performance Common wisdom suggests that people who perform best in simulations do so because they have learned how to play the game better than most. And, Anderson and Lawton (1990) reported that 92.5% of simulation users in colleges surveyed used financial performance in the simulation as a determinant of a students grade. This implies that most simulation users behave performance influences whatever grades reflect (which presumably is learning). However many authors contend that the relationship between performance in the simulation and learning from it is weak or nonexistent. Greenlaw and Wyman (1973) and Thorngate and Carroll (1987) argue that simulation performance is due to luck (and therefore not due to learning). Burns, Gentry and Wolfe (1990) explain that performance can 43 Developments In Business Simulation & Experiential Exercises, Volume 23, 1996 be affected by luck or other players performing poorly while learning Is the internalization of rules which might occur as a consequence of mistakes. Other than the studies done by the present authors, only three previous studies deal directly with the relationship between learning and performance. Teach (1989) found that forecasting accuracy (for him a measure of learning) did correlate significantly with measures of profits. As mentioned above, Wellington and Faria (1991) found no relationship between level of simulation performance and exam scores. Also, Anderson and Lawton (1992) found that only two of seven learning measures correlated with financial performance. These two were played down because both involved the comprehensiveness and workability of annual plans. Thus the authors concluded a paucity of significant results. To summarize the previous literature on the correlates of learning, there is little in the literature that suggests why some learn more than others. There is evidence indicating that learning and performance do not covary, but there is little suggesting variables that do correlate with or predict learning in the simulation. The Present Series of Studies Our three previous studies (Wash bush and Gosenpud, 1993, 1994, & 1995) have all had purposes similar to the present effort. In each of the studies, correlations between learning and performance have been insignificant and near enough to zero so that we can comfortably conclude that there is no relationship between learning and performance at least for University of Wisconsin-Whitewater policy students. Regarding other variables in our 1993 effort, we found that learning was greater for members of teams that were either 1) in the middle of a competitive race to attain or maintain simulation standing or 2) improving in position. Learning was less substantial for teams that faced less competition and teams whose competitive standing was declining. We hypothesized that those who were trying or struggling would learn more and those not trying or coasting would learn less and tested this hypothesis in our 1994 and 1995 studies. The results show no relationship between struggle and learning. In addition in the 1994 and 1995 studies (unlike the results from 1993), there was no tendency for players either in competitive races or improving to learn more than players who were declining or coasting. The Present Study The present study is similar to the previous ones in that we measured whether a relationship exists between learning, on one hand, and performance and struggle on the other. In the present study, though, we proposed other variables that might predict the degree to which learning takes place. We proposed two sets of variables. The first set represents common sense reasons for why some individuals would learn more than others in any situation, and we propose two kinds of common sense reasons. The first kind lies in players’ perceptions of the simulation. For some the simulation might be easy or interesting, therefore they would learn more; while others might find the simulation boring or difficult and therefore learn less. The second kind of reasons might lie in the emotional state of the individual. Some may be secure, motivated, confident, or improving and therefore learn more; while others might be afraid, unmotivated, not confident or declining and therefore learn less. The second set of predictor variables involves goals. There are a few studies (Gosenpud, Miesing and Milton, 1984; Hornaday and Curran, 1988; and Curran and Hornaday, 1990) that have focused on the relationship between the degree of formal goal setting and simulation performance. The contention here is that the kinds of goals set may influence how much is learned In the simulation. So this study will attempt to ascertain if four kinds of variables influence learning performance, degree of struggle, variables commonly associated with learning such as confidence, and types of goals. METHOD Subjects, Research Design and Procedure The subjects of this study were 46 students enrolled in two sections of the required undergraduate Administrative Policy course at the University of Wisconsin-Whitewater during the Fall 1994 semester. Each section compiled an industry, The Micromatic simulation (Scott et. al., 1992) was used in both, and both were taught by the senior author of this paper. The simulation length was 11 quarters in one industry and 13 in the other. Both industries were identical with respect to decision factor weights. Simulation performance was based on Net Income (40%), Return on Sales (30%) and Return on Assets (30%). The game was worth 20% of the course grade; 5% of the course grade was based on peer ratings of team contribution; 5% of the course grade reflected the score on an exam measuring learning in the simulation. Learning To measure learning, we developed two forms of a multiple- choice and short-essay examination. These forms were made deliberately parallel in form and content. The examinations were constructed using 44 Developments In Business Simulation & Experiential Exercises, Volume 23, 1996 questions and situations routinely confronted by companies competing in Micromatic. These included manipulating and analyzing the marketing-mix, making operating decisions, determining costs of goods sold, understanding the consequences of doing or not doing ratio analysis or cash flows, and understanding the relationship between plant capacity and marketing expenses. The questions tapped analysis, synthesis, and application skills of the Bloom Taxonomy (Bloom, 1956). For all studies, Form I was administered as a pre-test at the beginning of the semester. Form 2 was administered at the end of the semester. Learning over the period of play was defined as the percentage score for Form 2 minus the percentage score for Form 1. The test developers used a common scoring key for all questions to ensure uniformity of measurement. Statistical reliability estimates for the instruments range from .65 to .7. Other Variables Struggle, goals and the common sense variables were measured three times, after the third quarter, as the results of the sixth quarter were returned to the students, and after the tenth quarter. The degree to which students were struggling was measured by two Likert-type questions, goals were obtained from an open-ended question, and the common sense variable information was obtained with fifteen bi-polar semantic differential items. Simulation performance was measured using the scoring routine in the Micromatic software. RESULTS Learning and performance Learning scores correlated -.13 with performance at quarter 3, -.07 with performance at quarter 7, -.21 with performance at quarter 9 and .12 with end-of-game performance. None of these correlations were significant, suggesting that learning and performance do not co-vary. Struggle and common sense variables Table 1 shows the correlations between learning and the hypothesized continuous predictor variables Including the two struggle and the fifteen common sense variables. It shows learning did not correlate with whether or not individuals struggled with the simulation. Learning did correlate with some of the common sense variables, In particular those reflecting how well individuals seemed to understand the simulation early on. Degree of learning correlated significantly with how well students felt they understood the simulation at quarters three and seven and how simple they thought the simulation to be at quarter three. Although not significant, there was also a slight tendency for students who saw themselves as Improving to learn more. TABLE 1: CORRELATIONS WITH LEARNING Qtr 3 Qtr 7 End of Game STRUGGLE RELATED Struggling .02 -.16 -.11 Not Struggling .00 .21 .00 COMMON SENSE Threatening (intriguing) -.11 -.10 -.05 Challenging .16 -.24 .15 Improving -.11 -.21 -.09 Scattered (Consistent) -.17 -.07 .22 Falling -.01 .06 .20 Safe .06 .10 -.03 Hopeless .00 -.14 .03 Rewarding .13 -.17 -.02 Positive -.03 .10 -.06 Simple .32* -.23 .17 Irrelevant -.23 -.16 -.09 Regressing (improving) -.27 -.17 -.01 Inert -.27 -.23 .05 Confused (Understood) -.33* -.31* .13 Stimulating .15 .11 .01 * p less than .005 Goals Table 2 shows learning scores as a function of expressed goals at various stages of the game. The mean learning score for the sample was .02, which means that the sample as a whole-averaged 2% better on the post test than on the pre- test. Table 2 shows that learning scores for students expressing certain goals were considerably higher than the overall mean, Learning scores were at least somewhat lower for students expressing other goals. Simulation learning was higher when learning, stock price, profit and expansion- related goals were stated, and lower when turn-around and competitive goals were stipulated. None of these differences were significant. 45 Developments In Business Simulation & Experiential Exercises, Volume 23, 1996 TABLE 2: LEARNING SCORE IF GOAL STATED Qtr 3 Qtr 7 End of Game Mean N Mean N Mean N LEARNING RELATED Expand knowledge .013 6 .020 2 .040 Learn Decision Making .012 5 Make Good Decisions .037 3 .040 1 Learn from Mistakes .070 2 Gain Group Experience .040 1 .040 1 .040 1 Integrate Business Functions .085 2 .075 2 Total .026 11 .035 6 .040 1 COMPETITVE GOALS Do Well (Comparatively) .008 5 .018 4 .046 8 Not Finish Last -.010 1 Win .014 5 -.032 5 -.010 1 Place in Top 2 -.038 4 .030 4 -.004 7 Move Up in Rankings .043 4 .010 1 Stay at or Near Top -.005 2 -.003 11 Improve -.017 -.010 1 -.008 8 Total -.008 17 .010 13 .009 23 PROFIT RELATED Make Profits .052 11 .019 8 .004 10 Maximize Return on Sales .005 1 Reduce Expenses .076 5 .038 6 .056 5 Match Capacity & Demand -.010 1 -.100 1 -.025 2 Get out of Debt -.050 1 Total .048 13 .010 13 .042 17 EXPANSION RELATED Expand .050 5 -.020 1 Increase Sales .070 4 .020 4 .067 3 Total .072 8 .012 5 .067 3 STOCK PRICE RELATED Maintain High Stock Price .032 5 .037 3 .260 2 Issue Stock .125 2 .018 4 .050 1 Total .032 5 .025 7 .103 3 GRADE RELATED Get and A -.070 1 .037 3 .080 2 Get Decent Grades -.120 3 Total -.070 1 .037 3 -.040 5 The results regarding expressed goals at quarter 3 seem meaningful. Students expressing goals associated with game related financial measures such as profits and sales early in the game seemed to have learned more. Those expressing competitive or grade-related goals learned less. Given that profits and sales are the ‘knitting of the game, then these results indicate that concentrating on the knitting early in the game helps one learn in it. 46 Developments In Business Simulation & Experiential Exercises, Volume 23, 1996 DISCUSSION As in past studies, there was a near zero correlation between learning and end of game performance. This solidifies the conclusion that there is no relationship between the two variables. The fact that there was a .2 correlation between learning and performance at the games three-quarter mark deserves a bit of notice. It is possible (but in our eyes not likely) that learning correlates with performance rankings or indices before the end of the game and not rankings or indices at the end of the game. The fact that for the third time, there was no relationship between degree of struggle and learning weakens our confidence that a relationship exists. Significant correlations did result between learning and simulation understandability and perceived simplicity of the simulation, early in its duration. There was also a tendency for leaning scores to be higher for those who set the goals in terms of financial indices instead of competitive or grade related ones. These two results suggest that the student’s early approach to the simulation influences learning. In our 1995 study (Washbush and Gosenpud, 1995), those who lacked confidence to a moderate (but not an extreme) degree learned the most. Combined, these results suggest that those who want to perform well financially early in the simulation, understand the simulation, but are not confident learn relatively more, while those who don’t understand the simulation and set competitive or grade-defined goals learn less. However, these statistical relationships are not strong and some are not significant, therefore we cannot be certain of these notions. They can serve as hypotheses for future research. There are problems with this series of studies. First we have measured learning as most professors do, by tests. We do not know whether they are valid. We have attempted to develop our tests to reflect the environment of the Micromatic simulation and test for learning goals suggested by experts (Gosenpud and Washbush, 1994). However we only have superficial evidence (reports from the test takers) that the tests measure learning in the simulation. Other ways to measure learning have been suggested and these have not yet been incorporated into the present series of studies. For example, Carvalho (1991) has developed a scheme to measure competence in the skills demanded by the simulation. In this scheme, if success demands increased marketing expenditure, learning is assumed to occur if players increase their marketing budgets with time. Petranek, Corey, and Black (1992) argue that the real learning comes from reflecting, especially when the learner writes. These authors quote Francis Bacon who said, “Reading maketh a full man...writing an exact man.” They report research on journal writing by Wollman-Bonilla (1989) who concluded that journals encourage understanding imaging, speculation questioning, and the shaping of Ideas. Second, our results are not compelling. We are only relatively certain of a negative result, that is that there is no relationship between learning and performance. We are still not clear as to what associates with and influences learning in the simulation. The variables we are working with are difficult to precisely define and measure. We believe that our efforts have yielded data suggesting researchable directions. We believe that simulation facilitators will gain greatly with information about variables that enhance learning. Our research to date has identified neighborhoods of variables to explore. But the precise variables and relationships have yet to be found. REFERENCES Anderson, RH & Lawton, L (1990) A survey of methods used for evaluating student performance on business simulations. Developments in Business Simulation & Experiential Exercises, 17, 177. Anderson, P.H. & Lawton, L. (1992) The relationship between financial performance and other measures of learning on a simulation exercise. Simulation & Gaming, 23, 326-340 Bloom, B. (ed.) (1956). The Taxonomy of Educational Objectives Handbook I: The Cognitive Domain. New York: David McKay. Bowen, D.D. (1987) Developing a personal theory of experiential learning: a dispatch from the trenches. Simulation & Games, 18, 192-206 Burns, A.C., Gentry J.W., & Wolfe, J. (1990). A cornucopia of considerations in evaluating the effectiveness of experiential pedagogies In J.W. Gentry (Ed), A Guide to Business Gaming and Experiential Learning East Brunswick NJ: Nichols/GP Brenenstuhl, D.C. & Catalanello, R.F. (1979) Can learning styles be used as curriculum aids. Journal of Experiential Learning and Simulation, 1, 29-37. Burns, A.C. & Sherrell, D.L. (1984). A path analytic study of the effects of alternative pedagogies. Developments in Business Simulation & Experiential Exercises, II, 115-119 47 Developments In Business Simulation & Experiential Exercises, Volume 23, 1996 Carvalho, G.F. (1991) Evaluating computerized business simulators for objective learning validity. Simulation & Gaming, 22, 328-348 Curran, K.E. & Hornaday, R.W. (1990) Formal planning and simulation team performance: A cross section approach. Developments in Business Simulation & Experiential Exercises, 17, 38-41 Faria, A.J. & Whiteley, T.R. (1990) An empirical evaluation of the pedagogical value of playing a simulation game in a principles of marketing course. Developments in Business Simulation & Experiential Exercises, 17, 53- 57 Gosenpud, J., Miesing, P., & Milton, C.J. (1984) A research study on the strategic decisions in a business simulation. Developments in Business Simulation & Experiential Exercises, 11, 161-165 Greenlaw, P.S. & Wyman, F.P. (1973). The teaching effectiveness of games in collegiate business courses. Simulating & Games, 4, 259-294 Hornaday, R.W. & Curran, K.E. (1988) Formal planning, simulation team performance, and satisfaction: A replication. Developments in Business Simulation & Experiential Exercises, 15, 138-146 Keys, B. (1976). A review of learning research in business gaming. Developments in Business Simulation & Experiential Exercises, 3, 25-32 Keys, B. & Wolfe, J. (1990). The role of management games and simulations in education and research Journal of Management, 16, 307-336 Petranek, C.F., Corey, S., & Black, R. (1992) Three levels of learning in simulations Simulation & Gaming, 23, 174- 185 Scott, T.W., Strickland, A.J., Hofmeister, D.L., & Thompson, M.D. (1992) Micromatic: A Strategic Management Simulation Boston: Houghton Mifflin Teach, R. (1989). Using forecasting accuracy as a measure of success in business simulations. Developments in Business Simulation & Experiential Exercises, 16,103- 107 Thorngate W. & Carroll, B. (1987) Why the best person rarely wins. Simulation & Games, 18, 299-320. Washbush, J. and Gosenpud, J. (1993) The relationship between total enterprise simulation performance and learning Developments in Business Simulation & Experiential Exercises, 20, 141. Washbush, J. and Gosenpud, J. (1994) Simulation performance and learning revisited. Developments in Business Simulation & Experiential Exercises, 21, 83- 85. Washbush, J. and Gosenpud, J (1995) Simulation performance, learning, and struggle Develop ments In Business Simulation & Experiential Exercises, 22, 1-4. Wellington, W.J. & Faria, A.J. (1991) An investigation of the relationship between simulation play, performance level, and recency of play and exam scores. Developments in Business Simulation & Experiential Exercises, 18, 111-115 Wellington, W.J. & Faria, A.J. (1992) An examination of the effect of team cohesion, player attitude, and performance expectations on simulation performance results. Developments in Business Simulation & Experiential Exercises, 19, 184-189 Whiteley, T.R. & Faria, A.J. (1989) A study of the relationship between student final exam performance and simulation game participation. Simulation & Games, 20,44-64. Wolfe, J. (1976. Correlates and measures of the external validity of computer-based business policy decision making environments. Developments In Business Simulation & Experiential Exercises, 7, 411-438. Wolfe, J. (1985) The teaching effectiveness of games in collegiate business courses: a 1973-1983 update. Simulation & Games, 16, 251-288 Wollman-Bonilla, J.E. (1989) Reading journals: invitations to participate in literature. The Reading Teacher, 112- 120 48 Table of Contents Volume 23, 1996 Modeling Advertising Effectiveness Simulation as an Aid to Learning: How Does Participation Influence the Process? Administering Business Simulations in Transitioning Economies: The Introduction of Simulation Gaming to Estonia Business Simulation Games: Current Usage Levels. A Ten Year Update The Relationship Between Interpersonal and Task Cohesiveness and Performance in a Business Simulation Game The Design of an ITS-Based Simulation: A New Epistemology for Learning Correlates of Learning in Simulations How Do We Know where we're going if we don't know where we have been: A Review of Business Simulation Research Making Cash Flow Come Alive and Sensible in the Classroom Enhancing Simulation Learning through Objectives and Decision Support Systems An Analysis of Deliberate and Emergent Strategies Relative to Porter's Generic Differentiator and Cost Leader: A Bias and Variance Modeling Approach Introducing Ethical Dilemmas into Computer-Based Simulation Exercises to Teach Business Ethics CEO Strategic Locus of Control Effects on Game Performance and Playing Behavior The Relational Database As a Link between Operations and Cost Accounting Goal Setting over Time in Simulations Computerized Business Simulations: A Workshop Exploring the Tutor's Role, Task & Needs Strategic Analysis of the Product Portfolio with the COMPLETE PPA Package: A Strategic Market Planning Tool Draft Standards and Registration Procedure for Assessment Instruments Perspectives on a New Generation of Business Games An Economic Multiple Regression Case In Experiential Learning Changing Institutional Norms and Behavior, Not Culture: Experiential Learning Comes to Myanmar Strategic Management and the Case Method: Survey and Evaluation Individual Differences in Internet Attitude and Use Long Live the Plan - or Should It? Examining the Impact of Detailed Strategic Plans on Organizational Performance Do Your Students Really Read the Manual? A Computerized Contextual Tool As A Surrogate for the Traditional Student Manual Pilot Analyses of Self-Peer Evaluations in an Experiential-Exercise Human Resources Management Course Leader Behavior Feedback: A Learning Exercise Dilemma-Dilemma: An Exercise for Teaching Significance Of Communication Computer Mediated Conferencing: Technology and Classroom Learning Multimedia in the Workplace: Who is really using it and where is it Headed? Using Experiential Exercises for Collecting Research Data: Integrating Teaching and Research Interactive Distance Learning as a Tool in a College's Theory and Practice The President's Decision: An Experiential Exercise in Decision Making Integrating Computer Literacy Skills in the Undergraduate Curriculum: The Advanced Accounting Experiment Imperatives for the Transfer of Experience-Based Training Deciding How to Decide The Internet as a Pedagogical Tool Internet Scavenger Hunt Two Management Exercises Based on Committee Work Multimedia in the Year 2000: How Will It Affect Our Lives? Bootstrap Benefit Segmentation: Finally A Way to Teach Benefit Segmentation without Primary Data or Those Fancy Statistical Methods Multimedia and Learning: Is There A Connection? A Changing Business Policy Collaborative Learning Through Real-Life Assignments in Accounting Classes The Necessity Of Incorporating Local Cultural Aspects Into International Business Experiential Exercises Utilizing Cultural and International Landmark Constructs to Assess Business Student's International Awareness Legal Issues related to the Use of Application Blanks: An Experiential Exercise The Family in the Classroom: An Experiential Exercise for Teaching Issues Related To Expatriate Assignments Using Internet Resources to Enhance Teaching of Information Systems Courses: A Demonstration Proposal Chalkboards to Chipboards for Teachers and Consultants How Do We Measure The Learning In Experiential Learning and How Do We Best Simulate It?