Why is Learning so Difficult to Measure when “Playing” Simulations Page 55 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 ABSTRACT The link between cognitive learning and success in business simulations has been elusive. Few research studies have successfully linked cognitive processes to business simulation outcomes with a significant amount of explained variance. This paper attempts to explain why this may be true and offers some suggestions to obtain better results. INTRODUCTION My grandpa, who only attained a 4th-grade education, once said to me, “You cannot learn how to ride a bicycle or how to swim, by reading a book,” and he was correct. (Grandfathers are almost always correct.) The grade on an examination after taking a course in economics, statistics, marketing, etc. can reveal most of what has been learned after the class its final examination. This knowledge may have been obtained during the class and its multiple assignments or before (or simultaneously) in a different course or even by experience during some undefined activity when the person was younger. All that we can certify is the amount of knowledge the student, at that moment in time, knows about the content of that specific course. Instructors practice this phenomenon every day. Why does this method not work well with business simulations? Joseph Wolfe et al. (2017) wrote “After over 60 years of business game usage in higher education, there is still no objective evidence that they teach a course’s subject matter. Many have attempted to answer this question, but few have succeeded” (Quote from the abstract). Let’s look at a typical business course. Each course designates a small body of specific knowledge. In microeconomics, it may be, “the price-demand curve slopes downward and to the right (in most cases), etc. In Statistics, it might be, a linear relationship between two variables is significant only if the “p” value is less than 0.05 and so on. (often a false assumption.) Most of what we teach are facts and methods; and it is assumed that given the facts and methods, the students should be able to apply them correctly. (There are numerous exceptions to this assumption.) Case courses require analysis and provide examples of alternative solutions, or try to point toward an acceptable solution with home-work providing practice. Simulations require analysis of situations, decisions necessary to be competitive, living with the consequences and planning for the next and future periods. The reasoning behind using a business simulation in a class is simply to have the participants get better and better at analysis, decision- making, and planning, round after round. According to Kolb (1984). “Learning is the process whereby knowledge is created through the transformation of experience.” He identified 4 steps of experiential learning, these being: “1) concrete experience (CE); 2) Reflective observation (RO); 3) Abstract conceptualization (AC); and 4) Active experimentation (AE). Active experimentation and concrete experience together constitute the tangible dimension of learning.” These four steps describe how most business simulations work in a business school setting. HOW HAS ABSEL APPROACHED THIS PROBLEM IN THE PAST ABSEL authors have taken three approaches to the issue of learning from simulations. These have been: 1. Attempting to determine if any learning takes place as the result of participating in a business simulation. 2. Attempting to determine the relationship between simulation performance and various measures of cognitive learning. 3. Attempting to determine what learning may have occurred during the process of participating in a business simulation. This research included both cognitive as well as non-cognitive learning. Does Any Learning Take Place? At the very first ABSEL conference, Achilles Armenakis et al. (1974) reported a study of the performance of teams participating in the 1974 Emory University Intercollegiate Business Game. The authors devised a scoring system and evenly split the teams into two groups. They then attempted to determine the differences between the best performing teams and the poorer performing teams. That study had four primary hypotheses: 1. Do successful teams employ more quantitative methods than unsuccessful ones? No statically significant differences were found. Why is Learning so Difficult to Measure when “Playing” Simulations Richard Teach Professor Emeritus, the Scheller College of Business at Georgia Tech richard.teach@scheller.gatech.edu Page 56 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 2. Does the experience in participating in the I.B.G. distinguish between teams that are successful and unsuccessful? No statically significant differences were found. Competing in a previous Emory University business simulation contest did not enhance the chances of winning a later Emery contest. 3. Does previous experience participating in business games differentiate successful and unsuccessful teams? A significant difference was found with (“p”) < 0.05). This is an important finding, as it indicates that participants had learned from previous “plays” of a business simulation. 4. Do successful teams devote more time to making decisions than unsuccessful teams? No statically significant differences were found. In the 3rd ABSEL conference, David J. Fritzsche (1976), Reported on learning styles, perceived learning, and simulation performance, using game grades as the dependent measure. His learning styles were non-cognitive measures defined by Kolb and Goldman (1973). These learning styles were identified as divergers, assimilators, convergers, and the accommodators. The meanings of these four styles are: “The Diverger ...is best at Concrete Experience (CE) and Reflective Observation (RO). His/her greatest strength lies in his Imaginative ability.” “The Assimilator’s dominant learning abilities are Abstract Conceptualization (AC) and Reflective Observation (RO). His/ her greatest strength lies in his ability to create theoretical models.” “The Converger’s dominant learning abilities are Abstract Conceptualization (AC) and Active Experimentation (AE). His/ her greatest strength lies in the practical application of ideas.” “The Accommodator …has the opposite strengths of the Assimilator. He/she is best at Concrete Experience (CE) and Active Experimentation (AE). His/her greatest strength lies in doing things, in carrying out plans and experiments and involving himself in new experiences” (Quote from page 457, Fritzsche 1976). The hypothesis that the grades assigned to the simulation differed depending on the learning styles was rejected with “p” = 0.25. Miles, Biggs, and Schubert (1986) reviewed sixteen studies that used student self-judgment of skill acquisition through cases and simulation games as the dependent variable. They concluded that the mixed results uncovered were difficult to interpret and compare because of the wide variety of study environments used. THE RELATIONSHIP BETWEEN SIMULATION PERFORMANCE AND COGNITIVE LEARNING Whiteley and Faria (1989) conducted a study attempting to measure the learning from a business simulation by the final exam performance. Students were divided into two groups after the midterm exam., one playing a simulation and the other group did not. Both groups attended the post-midterm lectures in the class. The assumption was that those who “played” the simulation would perform better on the final after taking the effect of the midterm exam into consideration. The effect of the midterm on the final grade was significant (“p” <0.001) but the effect of the gameplay was not. They reported (“p”) > 0.05). Their remark was “Despite the proliferation, and widespread use of business simulation games, a review of the literature reveals that the pedagogical value of such games remains unclear.” (Quote from page 78) Teach (1990), claimed profit was a false prophet of learning. He suggested that “Better measures of managerial ability would be gained by measuring and analyzing errors in forecasting over a wide variety of events. The ability to operate within budget constraints and to allocate limited resources among almost limitless needs is also an indicator of managerial ability.” “Measuring profit performance requires the limitation that all firms must start as equals.” (The quotes are from the abstract.) Washbush and Gosenpud (1993), examined the relationship between learning and simulation performance, using a total enterprise simulation. They “found no direct, positive linear correlations between learning and simulation performance” (Quote from page 141). In the following year, Washbush and Gosenpud (1984) restudied that relationship. This time they divided the teams into three groups, a top performing set of teams, a middle-of-the-road set of teams and a set of the poorest performing teams. Voila! A strong relationship exists between the middle-of-the-road group. (“p” < 0.005) But no significant findings for the leaders or those with poor performing teams. The hypothesized was that teams that are leading have such a comfortable lead that they no longer have to work. Teams at the back-of-the-pack lost hope and gave up. Middle-of-the-roaders see chances of winning if they work hard. Long-term leaders have a substantial advantage over the rest of the teams may contribute to this “giving-up” phenomenon. (see Patz (1999, 2000, 2001, 2002. 2003 and 2004) and Teach and Patel (2007). LEARNING BY PLAYING Gee (2003) wrote “…the learning principles that good games incorporate are all strongly supported by contemporary research in cognitive sciences - the science that studies human thinking and learning through laboratory research, studies of the brain, and research at actual learning sites like classrooms and workplaces. Gee went on saying beyond using the learning principles that good games incorporate, I also argue that schools, workplaces, and families can use games and game technologies to enhance learning. Further, I believe that use of games and game technologies for learning content in schools and skills in workplaces will become pervasive.” Page 57 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 Effect of Complexity Examining the degree of learning by a simulation participant, this outcome is affected by the complexity of the simulation. Numerous authors have reported a variety of views. Complexity and realism may not be an advantage. Springer et al. (1965) wrote that “The power of a model in solving a problem comes precisely from it's not corresponding to reality except in those details pertinent to the problem at hand.” (Quote from page 178). Frazer (1977) wrote, “...playing many different (simple) games in a course with students on different teams for each game is now a viable alternative to the traditional management simulation played over a prolonged period.” (Quote from page 3). Wolfe (1978) conducted a study to investigate the link between game complexity and the acquisition of business policy knowledge. Wolfe reported, “The simple game increased knowledge in two of the emphasis areas - the need for reappraisal and flexibility, and the effects of individual and group factors in policy and decision-making situation. The IG [intermediate complexity game] improved only one area, while the CG [complex game] improved a player’s knowledge in all five areas.” (Quote from page 149). Wolfe’s selection of using only two simulations to represent complexity may have resulted from the difference between the two simulations rather the difference in complexity. Patz (1990, 1992, 1995, 1999, 2000, 2001, 2002, 2003, 2004, 2005 and 2006) noted problems when evaluating participants in several total enterprise simulations. The participant biased evaluations were by a phenomenon he called “Dominance,” which occurs when teams whose firms have the best performance early in the game seem to obtain a great deal of market power and go on to maintain or dominate the industry regarding performance. The results of Professor Patz’s results were composite measures of firm performance which often have as many as ten identifiable measures. Hall and Cox (1994) challenged the assumption that complexity was necessary to create educationally effective business simulations. These authors described two aspects of realism. The first assertion was that realism was “a key determinant of educational effectiveness and that realism was produced through complexity.” And the second aspect was that “the amount of cognitive processing performed by participants relate to the simulation’s complexity. In turn, the simulation’s duration relative to cognitive processing produces cognitive pressure that may lead to role overload.” This role overload was negatively related to the level of adult learning (Quotes from page 30). The risk of role overload (French and Caplan 1972) is very great in large-scale simulations. TTeach (1990b) identified this problem as analysis paralysis. Teach and Murff (2008), reviewed complexity and its effects on simulation performance and learning. They suggested that the suggested preparation time and playing time as effective measures of simulation complexity. TABLE 1 A SHORT LIST OF WHAT STUDENTS MIGHT LEARN WHILE PARTICIPATING IN A BUSINESS SIMULATION 1. How forces outside of the firm's control may affect the firm’s performance 2. How to understand marginal analysis 3. How to understand Opportunity costs 4. The importance of the many topics of forecasting and the costs of forecasting errors 5. What unintended consequences are 6. How product life cycles affect decisions 7. How variable costs turn into fixed costs as soon as commitments are made 8. The importance of product positioning 9. How to work in teams 10. How to differentiate important information from unimportant information 11. How to work under uncertainty 12. How to determine interactions among two or more decision variables 13. How to anticipate competitive responses 14. How to considering possible competitors’ decisions when proposing strategies 15. How to analyze reports and financial results 16. How to assess risk 17. How to be innovative 18. How to be creative 19. How to create budgets 20. How to interpret useful statistics Page 58 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 ARE NON-COGNITIVE VARIABLES A DETERMENT OF SUCCESS: A SURPRISE FINDING? No, not at all. We have all noticed highly successful entrepreneurs who have either never attended a college or university or have dropped out of college or university and started a successful business. In spite of business schools’ claims to train future business executives, more CEOs and senior management personnel do not have MBAs than those who have these prestigious degrees. There is much more to learning than the cognitive sciences. How and why does this phenomenon occur? One big problem found in studies that want to link the attributes of participating in business simulations to learning lies in degrees of freedom. Too many variables, too few observations, as well as a lot of unobservable, non-cognitive learning (skills) of the team members, confuses the outcomes and muddies the results. What is the student expected to learn while participating in a business simulation? Some simple questions come to mind. “What price should I set, when I do not know what my competitors are going to do?” Or, “How do I forecast next period’s industry level of demand.” Or. “Once an estimate of industry demand is forecast, how do we then estimate our own firm’s unit demand to set our production schedules that will reduce the out-of-stock costs?” These are only three of the important things a student needs to learn to be successful in a business simulation. There are many others. Then why can we not determine an individual’s learning by using a multiple regression of the estimated causes of forecasting errors? (Teach, 2006) Does this measure learning? Table 1 shows a brief and incomplete list of what simulation participants might need to learn while participating in a business simulation. The table is only a small example. Almost any person who has worked with business simulations is capable of producing a list of 20 to 40 items that students might learn while participating in a competitive simulation. Almost every item on the above list has been taught in Business school classes, and most students can explain how each term is defined, but most do not know how, when or where to apply the concepts. The how, when and where may be learned most efficiently through the adoption of experiential exercises such as business simulations in a business school curriculum. Let’s face it, most college-level business courses teach facts and methodology, and not how to implement methods nor how to determine which facts to use. Simulations rely on skills needed to interpret which facts are more important and what facts are less important. To successfully compete in a simulation, students need the skill of understanding the boundaries of uncertainty. (i.e., Is the student able to estimate what price a competitor might be charging in period t+2? These are skills, not raw knowledge but competencies. Both leadership competencies and functional competencies are needed. GRIT A research study by Tim Kautz et al. (2008) provides some insight into measuring skill levels. The Kautz study reported “… non-cognitive skills such as perseverance skills (“grit”), conscientiousness, self-control, trust, attentiveness, self-esteem and self- efficacy, resilience to adversity, openness to experience, empathy, humility, tolerance of diverse opinions and the ability to engage productively in society, which are valued in the labor markets, in school, and in society at large.” (Quote from page 9.) “Achievement test scores predict only a small fraction of the variance in later-life success.” Heckman et al. (2014) found achievement tests explained only 17% of the variability of later-life earnings. Heckman and Kautz (2014) found IQ tests alone explained only 7 percent of the variability in later-life earnings. The author considers that classroom examinations could be included under the rubric of “achievement tests.” Given this observation, why would success in university coursework predict success in running a simulated business? It requires much more than standard academics. It seems that the term “non-cognitive skills” when combined with achievement scores such as grades, may better define success than academic knowledge alone and could be predictive of success in business simulations. “Skills are not set in stone at birth and determined at birth and determined solely by genes. They can be fostered. Cognitive and non-cognitive skills change with age and instruction.” … “non-cognitive skills are more malleable at later ages than cognitive skills (Quote from page 10). “Many psychologists do not recognize the differences between these measures and interchangeably use IQ, achievement tests, and grades to measure “cognitive ability” or “intelligence” and this practice is also wide-spread in economics” (Benjamin et al. 2013). The term “skills” indicates that these skills can be taught and learned. Skills have been widely studied, and there is a taxonomy of non-cognitive skills called the Big 5. (Borghans et al. (2008). These are Openness to Experience, Consciousness, Extraversion, Agreeableness, and Neuroticism/Emotional Stability. Duckworth et al. (2007) defined an additional non-cognitive skill they called “Grit,” which is a measure of persistence on task or in the pursuit of a goal. Previous research has provided evidence of the predictive validity of the Grit Scale on educational attainment among adults, and grade point average (GPA) among Ivy League undergraduates (See Duckworth et al. 2007). Lora Reed introduced the concept of Grit to ABSEL 2016. She commented “Grit may be more important for academic success than innate intelligence. Grit consists of traits including ‘resilience, self-control, and persistence’ (Pappano, 2013, p. 4). ‘It doesn’t depend just on what is inside of us but on being in the right circumstance with people we trust.” Page 59 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 According to authors Goodwin and Miller (2013), grit consists of four components; 1) Directedness: A person must begin with a goal in mind, be intent on accomplishing that goal, and must dedicate time and talent to attain the objective or end state. 2) Motivation: (Fenton, 2015); both intrinsic and extrinsic. Motivation means that the person has the desire or passion for achieving the goal, and the goal is meaningful to that person. 3 Self-control: Self-control requires an individual to avoid distractions and remain focused on the task (Goodwin and Miller, 2013). 4) A positive mindset: A positive mindset consists of embracing challenges and accepting failure as part of learning. Directedness or the pursuit of a goal may require long hours of tedious work, may take priority over other activities, and may necessitate the denial of pleasure (Shoda, Mischel, and Peake, 1990). Given the amount of focus and labor involved, the person must decide that the goal is worth pursuing. In other words, a person cannot wander aimlessly through life, lackadaisically derive a plan, and yet expect a lifelong dream to come to fruition. Grit requires commitment and performance; a desire to succeed without work is just desire. Furthermore, some research has shown there is an inverse association between grit and an orientation toward pleasure (Suzuki, Tamesue, Asahi, and Ishikawa, 2015). While no one expects total deprivation, individuals with grit are willing to forego pleasure or gratification to continue working towards their goals. This mindset of goal-over-pleasure is a significant part of what puts gritty people into a league of their own. The second component of grit is motivation. If a person has the desire to achieve his or her goal, and that goal is meaningful, then motivation exists. According to the American Psychological Association (APA), there are two types of motivation; the key difference is whether the motivation arises from the outside (extrinsic) or the inside (intrinsic). Extrinsic motivation occurs when a person expects to be rewarded, receive praise, or evade punishment and chastisement (Cherry, 2013). A simple view of extrinsic motivation is that it has two subsets: (1) the person receives recognition or (2) he eludes adversity. In both situations, the individual behavior is rewarded. “With intrinsic motivation, a person performs a task because he/she finds the activity personally rewarding” (Cherry, 2016). In this case, the person does seek a reward from others. “Intrinsic motivation has two subsets: (1) the individual enjoys the task on a personal level or (2) despite dislike or disinterest, the individual wants to solve the problem or overcome the challenge. In either case, the desire is innate.” The achievement of a goal is an intrinsic and extrinsic motivation. “However, in dealing with grit, extrinsic motivation is not enough; the goal is personal and the achievement of that goal requires intrinsic motivation,” Fenton (2015). “Students tend to enjoy learning and to do better when they are more intrinsically rather than extrinsically motivated to achieve” (Lucariello et al. 2016 Principle 9). Self-control results in better adjustment, higher self-esteem, better interpersonal skills and more optimal emotional responses (Tangney 2004). “Self-control refers to the capacity for altering one's responses, especially to bring them into line with standards such as ideals, values, morals, and social expectations, and to support the pursuit of long-term goals.” (Baumeister et al. 2007 Quote from the abstract) “The fourth component is a positive mind-set. A positive mind-set consists of embracing challenges and accepting failure as part of learning” (Goodwin and Miller, 2013). Challenges and failure are only one part of life. “While obstacles are ubiquitous, it is a person’s response to problems and failure that separates her from the fold. Stress-response is as individual as a person’s immune system; analogously, some people will succumb to infection whereas others will remain healthy. Likewise, some people will face challenges with bravado; others will shy away. Avoidance is not necessarily a bad thing; avoidance is a part of evolution in which humans recognize they need to avoid conflict, ambiguity, or hardship because the circumstances threaten their survival,” (Nicholson, 1998). Some situations merit avoidance, but gritty people are more willing to accept challenges and take the appropriate risk to achieve their goals. Reed and Jeremiah (2017) put it succinctly “Grit-in-short, although grit is not the only predictor of student success in both personal and professional contexts, it is certainly one predictor that should not be overlooked.” (Quote from page 225) There is a scale for measuring “Grit,” developed by Duckworth et al. (2007). That scale has two dimensions; Consistency of Interests and Perseverance of Effort each with six variables, discovered with Common Factor Analysis using Promax Rotation. The Big 5 Non-Cognitive Skills Psychologists primarily measure non-cognitive skills by using self-reported surveys or observer reports. They have arrived at a relatively well-accepted taxonomy of non-cognitive skills called the Big Five, with the acronym OCEAN, which stands for Openness to Experience, Conscientiousness, Extraversion, Agreeableness, and Emotional Stability. (Goldberg 1981) Openness to Experience is a general appreciation for art, emotion, adventure, unusual ideas, imagination, curiosity, and variety of experience (Goldberg 1993) & (McCrae & Costa, 1987). Conscientiousness is a tendency to display self-discipline, act dutifully, and strive for achievement against measures or outside expectations (Costa & McCrae (1992). Extraversion, or happiness Extraversion, focuses primarily on the quantity and intensity of relationships. (DeNeve and Cooper 1998) http://journals.sagepub.com/author/Baumeister%2C+Roy+F https://en.wikipedia.org/wiki/Conscientiousness Page 60 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 Agreeableness (or Pleasantness): A tendency to be compassionate and cooperative rather than suspicious and antagonistic towards others (Toegel and Barsoux 2012). Emotional Stability: The tendency to experience negative emotions, such as anger, anxiety, or depression. (Norris, Larsen, and Cacioppo 2007) and (Goldberg 1992) Originally, the Big Five was determined by having subjects self-report on 100 characteristics using Likert scale responses (Disagree strongly; Disagree a little; Neither agree or disagree; Agree a little; and Agree strongly) to questions such as: “I see myself as someone who: followed by 100 short statements like “is talkative”, and “tend to find fault with others.” Then in 2007, Beatrice Rammstedt and Oliver John produced a more manageable scale using only ten items. “It is believed that the Big Five traits are predictors of future performance outcomes. Job outcome measures include job and training proficiency and personal data.” (Mount & Barrick 1998). If these five traits are indicators of future performance outcomes, they may be predictors of business simulation outcomes AND these traits may be measurable both before and after “playing” simulations to see if any or all of them may be enhanced by participating in business simulations. Predictive Power of Grit and The Big Five Table 2. shows the decomposition of the variance for achievement and grades using IQ scores to represent cognitive ability, the combined set of the Big 5 non-cognitive skills and measures of Grit as independent variables. (Heckman et al. 2014) If all three variables were completely independent of one another, the maximum explained variance would equal 23%. Thus, these variables are almost orthogonal with each supplying important information. The dependent variable for this study was the score on an achievement test. The author believes that one may find even stronger relationships if a “success in simulations score” would be the dependent variable. How could non-cognitive skills be taught? Just like riding a bicycle, swimming and getting to Carnegie Hall, performance is enhanced by “practice, practice, practice.” The parallel being; simulations played in rounds are practice. “Skills” suggests that these attributes can be learned. In reality, the extent to which these personal attributes can change lies on a spectrum. Both cognitive and character skills can be changed over one’s life cycle, but through different mechanisms and with different ease at different ages. (Heckman and Kautz 2013). WHERE DO WE GO FROM HERE? The next step is to have researchers to use the scales for Grit and the Big Five to create profiles of those participating in business simulations and discover if this concept is, in fact, valid. The author suggests that Grit and the Big Five may be more predictive than the cognitive measures we have used in the past. If one uses before and after measures when business simulations in the classrooms, it might discover that the repetitive nature business simulations enhance both Grit and the Big Five scores. TABLE 2 THE PERCENT OF EXPLAINED VARIANCE OF ACHIEVEMENT SCORES WHEN USING IQ, THE BIG 5 NON-COGNITIVE SKILLS AND GRIT AS INDEPENDENT VARIABLES https://en.wikipedia.org/wiki/Compassionate https://en.wikipedia.org/wiki/Cooperative https://en.wikipedia.org/wiki/Paranoia https://en.wiktionary.org/wiki/antagonism http://sloanreview.mit.edu/article/how-to-become-a-better-leader/#article-authors--outside https://en.wikipedia.org/wiki/Neuroticism Page 61 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 REFERENCES Armenakis, Achilles, Hubert Feud, and William Holley, (1974) “Correlates of satisfaction, learning and success in business gaming.” Simulations, Games and Experiential Learning Techniques, Vol. 1, 1974, pp 272 to 278. Roy F. Baumeister Roy F., Kathleen D. Vohs and Dianne M. Tice (2007), The Strength Model of Self-Control, Current Directions in Psychological Science, Vol. 16, No. 6, pp 351 to 355 Benjamin, Daniel J., Sebastian A. Brown and Jesse M. Shapiro (2013), “Who is ‘behavior?’ cognitive ability and anomalous preferences.” Journal of the European Economic Association, Vol. 11, Issue 6, pages 1231 to 1255, Borghans, Lex, Angela Lee Duckworth, James J. Heckman and Bas ter Weel, (2008, Fall). “The economics and psychology of personality traits,” Journal of Human Resources Vol. 43. No. 4 pp 972 to 1059 Burns, Jeanne M. and Alvin C. Burns (1990), An Exposition of Guilford’s SJ Mode’ as a Means of Diagnosing and Generating Pedagogical Strategies in Collegiate Business Education,” in John Wingender and Walt Wheatley (Eds.), Developments in Business Simulation and Experiential Exercises, Vol. 17, pp. 34 to 37. Cherry, Kendra (2016). “Differences between Extrinsic and Intrinsic Motivation.” Retrieved from https:// www.verywell.com/differences-between-extrinsic-and- intrinsic-motivation-2795384 Costa, Paul T., and Robert R. McCrae (1992), "Four ways five factors are basic." Personality and individual differences 13.6 (1992): 653-665. DeNeve, Kristina M. and Harris Cooper (1998), “The Happy Personality: A Meta-Analysis of 135 Personality Traits and Subjective Well-Being,” Psychology Bulletin. Vol.124, No. 2, pp 197 to 229 Duckworth, Angela, Christopher Peterson, Michael Matthews, and Denis Kelly (2007), “Grit: Perseverance and passion for long-term goals,” Journal of Personality and Social Psychology, 92(6), 1087-1101. Duckworth, Angela. (2013, April), “The key to success?” Grit. TED Talks Education. Retrieved 3/2/2015 from http:// www.ted.com/talks/ angela_lee_duckworth_the_key_to_success_grit Duckworth, Angela, Christopher Peterson, Michael D. Matthews and Dennis R. Kelly (2007), “Grit: Perseverance and Passion for Long-Term Goals,” Journal of Personality and Social Psychology, 2007, Vol. 92, No. 6, 1087 to 1101 Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). “Grit: Perseverance and passion for long- term goals,” Journal of Personality and Social Psychology, Vol. 92, No, 6 pp 1087 to 1101. Fenton, Melissa S. (2015), "Exploring the Relationship Between Parental Psychological Control and Emergent Leadership." Theses, Dissertations, & Student Scholarship: Agricultural Leadership, Education & Communication Department. 102. http:// digitalcommons.unl.edu/aglecdiss/102 Frazer, J Ronald (1977), “Time-sharing business games,” Developments in Business Simulations and Experiential Exercises, Vol. 3, pages 3 to 9. French, John R.B., Jr. and Robert D. Caplan (1972), French, J. R. P., R. D. Caplan, and A. Marrow. "The failure of success." New York. Fritzsche, David J. (1976), “On the relationships of learning style, perceived learning and performance in an experiential learning environment,” Computer Simulation and Learning Theory, Vol.3, 1977, pp. 455 to 462. Gee, James Paul (2003), “What Video Games Have to Teach Us About Learning and Literacy,” (New York: Palgrave/ Macmillan, 2003). Goldberg, Lewis R (1981), “Unconfounding situational attributions from uncertain, neutral and ambiguous ones: A psychometric analysis of descriptions of oneself and various types of others,” Journal of Personality and Social Psychology Vol. 41 pp 517 to 552 Goldberg, Lewis R. (1993). "The structure of phenotypic personality traits." American Psychologist. Vol. 48: pp 26 to 34. Goodwin, Bryan and Kirsten Miller (2013) ‘Research says evidence on flipped classrooms is still coming in”, Educational Leadership, vol. 70, no. 6 (Mar. 2013), pp. 78-80. Goldberg, Lewis R. (1981). “Language and individual differences: The search for universals in personality lexicons.” In L. Wheeler (Ed.), Review of personality and social psychology, (Vol. 2, pp. 141 to 165). Beverly Hills, CA: Sage. Goldberg, Lewis R. (1992). “The development of markers for the Big-Five factor structure,” Psychological Assessment, Vol. 47, pp 26 to 42. Goldberg, Lewis R. (1993), “The structure of phenotypic personality traits.” American Psychologist, Vol. 48, pp 26 to 34. Goodwin, Bryan and Kirsten Miller, (2013), “Grit+ talent= student success,” Educational Leadership, online at www, illinoiscte.org Gosenpud, Jerry and John Washbush (1994). “What simulation users think players should be learning from simulations,” Developments in Business Simulation & Experiential Exercises, Volume 21, pp 96 to 99 Hall, Jeremy S. B. and Benita M. Cox (1994), “Complexity, is it really that simple,” Developments in Business Simulations and Experiential Exercises, Vol. 21, pages 30 to 34. Harris, Janetta L. and Bobbie J. Murray (2017), “Educational Grit and Psychological Trauma,” Business Simulation and Experiential Learning, Vol. 44, pp 86 to Heckman, James J., and Tim Kautz (2013), “Fostering and measuring skills: Interventions that improve character and Cognition,” Working paper 19656, http// www.nber.org/papers/w19656. Heckman, James J., John. E. Humphries, and Tim Kautz (Eds.) (2014a). “The Myth of Achievement Tests: The GED and the Role of Character in American Life,” Chicago: University of Chicago Press. Harris, Janetta L. and Bobbie J. Murray (2017), “Educational Grit and Psychological Trauma,” Developments in Business Simulation and Experiential Learning, Volume 44, pp 86 to 94 http://journals.sagepub.com/author/Baumeister%2C+Roy+F http://journals.sagepub.com/author/Vohs%2C+Kathleen+D http://journals.sagepub.com/author/Tice%2C+Dianne+M http://journals.sagepub.com/author/Tice%2C+Dianne+M http://onlinelibrary.wiley.com/doi/10.1111/jeea.2013.11.issue-6/issuetoc https://www.verywell.com/differences-between-extrinsic-and-intrinsic-motivation-2795384 https://www.verywell.com/differences-between-extrinsic-and-intrinsic-motivation-2795384 https://www.verywell.com/differences-between-extrinsic-and-intrinsic-motivation-2795384 http://psycnet.apa.org/search/results?term=Peterson%2C%20Christopher&latSearchType=a http://psycnet.apa.org/search/results?term=Matthews%2C%20Michael%20D.&latSearchType=a https://www.dropbox.com/s/0y545gn2withb5e/DuckworthPetersonMatthewsKelly_2007_PerseveranceandPassion.pdf?dl=0 https://www.dropbox.com/s/0y545gn2withb5e/DuckworthPetersonMatthewsKelly_2007_PerseveranceandPassion.pdf?dl=0 https://www.dropbox.com/s/0y545gn2withb5e/DuckworthPetersonMatthewsKelly_2007_PerseveranceandPassion.pdf?dl=0 https://www.dropbox.com/s/0y545gn2withb5e/DuckworthPetersonMatthewsKelly_2007_PerseveranceandPassion.pdf?dl=0 http://digitalcommons.unl.edu/aglecdiss/102 http://digitalcommons.unl.edu/aglecdiss/102 http://dayschool.firstpresclarksdale.com/ucm/school/adbar_pdfs/adbar_1.pdf http://dayschool.firstpresclarksdale.com/ucm/school/adbar_pdfs/adbar_1.pdf Page 62 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 Kautz, Tim, James J Heckman, Ron Diris, Bas ter Weel and Lex Borghans (2008) Fostering and Measuring Skills: Improving cognitive and non-cognitive skills to promote lifetime success. OECC (Online) Klein, Ronald (1980), “Can business games effectively teach business concepts?” Experiential Learning Enters the Eighties, Vol.7, pp 128 to 131 Kolb, David A., (1984), “Experiential learning Experience as the source of learning,” Englewood Cliffs, NJ: Prentice-Hall. Kolb, David A. and Goldman, Marshall B., (1973) “Toward a Typology of Learning Styles and Learning Environments: An Investigation of the Impact of Learning Styles and Discipline Demands on the Academic Performance, Social Adaptation and Career Choices of MIT Seniors,” M.I.T. Sloan School Working Paper #688-73, December 1973. Kolb, David A., (1984), “Experiential learning Experience as the source of learning,” Englewood Cliffs, NJ: Prentice-Hall. Laney, Marti Olsen (2002). The Introvert Advantage. Canada: Thomas Allen & Son Limited. pp.28 to 35. ISBN; 0- 7611-2369-5. Lucariello, Joan M., Bonnie K. Nastasi, Eric M. Anderman, Carol Dwyer, Heather Ormiston, and Russell Skiba. "Science supports education: The behavioral research base for psychology's top 20 principles for enhancing teaching and learning." Mind, Brain, and Education 10, no. 1 (2016): 55-67. Lena Silva (2008) McCrae Robert R. and Paul T. Costa, Jr (1987), “Validation of the Five-Factor Model of Personality Across Instruments and Observers,” Journal of Personality and Social Psychology, Vol. 52, No. 1, pp 81 to 90. Marrow, Alfred, J. (ed.) The Failure of Success (New York: AMACOM, 1972): page 3. Midkiff Brooke, Michelle Langer, Cynthia Demetriou, A. T. Panter (2017) “Measuring Grit Among First- Generation College Students: A Psychometric Analysis.” In: Andries van der Ark, Marie Wiberg, Steven A. Culpepper, Jeffrey A. Douglas, Wenyi Wen- Chung (eds.) Quantitative Psychology. Pp 407 to 420. Miles Jr, Wilford G., William D. Biggs, and James N. Schubert (1986), "Student perceptions of skill acquisition through cases and a general management simulation: A comparison." Simulation & Games Vol. 17. No.1, pp 7 -24. Mount, Michael K. and Murray R. Barric, (1998), “Five Reasons Why the ‘BIG FIVE’ article Has Been Frequently Cited,” Personnel Psychology, Vol.51: pp 849 to 857. doi:10.1111/j.1744-6570.1998.tb00743.x Nicholson, N. (1998). How Hardwired is Human Behavior? Harvard Business Review. Retrieved from https:// hbr.org/1998/07/how-hardwired-is-human-behavior Norris, Catherine J., Jeff T. Larsen, and John T.; Cacioppo, (2007) “Neuroticism is associated with larger and more prolonged electrodermal responses to emotionally evocative pictures,” Psychophysiology. 44 (5): pp 823 to 826. Oliver P. John and Sanjay Srivastava (1999), “The Big Five Trait Taxonomy: History, Measurement, and Theoretical Perspectives, Chapter 4 in L. A. Pervin, & O. P. John (Eds.), Handbook of personality: Theory and Research (2nded., pp.102-138). New York: Guilford Patz, Alan L. (1990), “Group personality composition and total enterprise simulation performance,” Develops in Business Simulation and Experiential Learning Vol. 16 pages 132 to 137. Patz, Alan L. (1992), “Confidence extremes diminish quality performance in a total enterprise simulation,” Develops in Business Simulation and Experiential Learning Vol. 19 pages 136 to 140. Patz, Alan L. (1995), “Revisiting personality bias in total enterprise simulations,” Develops in Business Simulation and Experiential Learning Vol. 22 pages 24 to 30. Patz, Alan, L. (1999), “Overall dominance in total enterprise simulation performance,” Develops in Business Simulation and Experiential Learning Vol. 26 pages 115 to 116. Patz, Alan, L. (2000), “One more time: Overall dominance in total enterprise simulation performance,” Develops in Business Simulation and Experiential Learning Vol. 27 pages 254 to 258. Patz, Alan, I. (2001), “Total enterprise simulation winners and losers: A preliminary study,” Develops in Business Simulation and Experiential Learning Vol. 28 pages 192 to 195. Patz, Alan, I. (2002), “Strategy learning in a total enterprise simulation,” Develops in Business Simulation and Experiential Learning Vol. 29 pages 143 to 148. Patz, Alan, I. (2003), “Revisiting strategy learning in a total enterprise simulation,” Develops in Business Simulation and Experiential Learning Vol. 30 pages 213 to 219. Patz, Alan, I. 2004), “Some strategies don’t learn or can’t learn,” Business Simulation and Experiential Learning Vol. 31pages. Patz, Alan, I. (2005), “When prophecy fails: A small sample, preliminary study,” Develops in Business Simulation and Experiential Learning Vol. 32 pages 240 to 243. Patz, Alan, I. (2006), “The BUSINESS STRATEGY GAME: A performance review on the online edition,” Develops in Business Simulation and Experiential Learning Vol. 33 pages 58 to 64 Pappano, Laura. (2013, January/February) ‘Grit’ and the new character education” Harvard Education Newsletter, Vol. 29, pp. 1 to 5. Rammstedt, Beatrice and Oliver John (2007), “Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German,” Journal of Research in Personality, Vol. 41 pp 203 to 212, Elsevier Reed, Lora and Jim Jeremiah, (2017), “Student grit as an important ingredient for academic and personal success” Developments in Business Simulation and Experiential Learning, Volume 44, 2017 pp 252 to 256 Reed, Lora, Marvee Marr and Alan Swank (2016), “Virtual Student Club Start-Up: Learning Together in Virtual Learning Community,” Developments in Business Simulation and Experiential Learning, Vol. 43, pp 292 to 295 Shechtman, Nicole., Angela DeBarger, Carolyn Dornsife, Soren Rosier, & Louise Yarnall. (2013). Promoting grit, tenacity, and perseverance: Critical factors for success in the 21st century. Washington, DC: U.S. Department of Education. https://en.wikipedia.org/wiki/International_Standard_Book_Number https://en.wikipedia.org/wiki/Special:BookSources/0-7611-2369-5 https://en.wikipedia.org/wiki/International_Standard_Book_Number https://en.wikipedia.org/wiki/Special:BookSources/0-7611-2369-5 https://hbr.org/1998/07/how-hardwired-is-human-behavior https://hbr.org/1998/07/how-hardwired-is-human-behavior Page 63 - Developments in Business Simulation and Experiential Learning, Volume 45, 2018 Shoda, Yuichi and Walter Mischel and Philip K. Peake (1990), “Predicting Adolescent Cognitive and Self-Regulatory Competencies: From Preschool Delay of Gratification: Identifying Diagnostic Condition,” Developmental Psychology Vol. 26, No. 6, pp 978 to 986 Silva, Elena (2008), “Measuring Skills for the 21st Century.” Education Sector Reports ERIC Number: ED503236, Record Type: Non-Journal, Publication Date: 2008- Nov-10 Pages: 18 Education Sector Springer, Clifford H, Robert E Herlihy and Robert I Beggs (1965), Advanced Methods and Models, Richard D. Irwin, Inc. Suzuki Yuzuki, Dai Tamesue, Kentaro Asahi, and Yoshiki Ishikawa (2015), “Grit and Work Engagement: A Cross-Sectional Study.” On-line at: https:// doi.org/10.1371/journal.pone.0137501 Tangney, June P., Roy F. Baumeister and Angie Luzio Boone (2004), “High Self-Control Predicts Good Adjustment, Less Pathology, Better Grades, and Interpersonal Success,” Journal of Personality Vol. 72, No. 2, (April) 271 pp to 324 Teach, Richard D. (1990a), "Designing business simulations," Guide to Business Gaming and Experiential Learning, (edited by James W. Gentry), Nichols/GP Publishing Teach, Richard D. (1990b), “Profits: the false prophet in business gaming,” Simulation & Gaming, Vol. 21, No. 1, pp 12 to 26. Teach, Richard (2006), “Forecasting Accuracy and Learning: The Key to Measuring Simulation Performance,” Developments in Business Simulation and Experiential Learning, Vol. 33, pp 48 to 57. Teach, Richard and Vishal Patel (2007), “Assessing participant learning in a business simulation,” Developments in Business Simulation and Experiential Learning, Vol. 34, pp 76 to 84. Teach, Richard and Elizabeth Murff (2008), “Are the Business Simulations We Play Too Complex?” Developments in Business Simulation and Experiential Learning, Vol. 35, pp 205 to 211. Toegel, Ginka and Jean-Louis Barsoux (2012). "How to become a better leader". MIT Sloan Management Review. Vol. 53 No.3, pp 51 to 60. Whiteley, T. Richard and Anthony J. Faria, (1989), “A study of the relationship between student final exam performance and simulation game participation,” Developments in Business Simulation & Experiential Exercises, Vol. 16, pp 78 to 83 Washbush, John and Jerry Gosenpud (1993), The relationship between total enterprise simulation performance and learning,” Developments in Business Simulation & Experiential Exercises, Vol. 20, p 141 Washbush, John and Jerry Gosenpud (1994), “Simulation performance and learning revisited,” Developments in Business Simulation & Experiential Exercises, Vol. 21, p 83 to 86 Wolfe, Joseph, (1978). The effects of game complexity on the acquisition of business policy knowledge. Decision Sciences, Vol. 9, No, 1) pp 143to155. Wolfe, Joseph, Robert Willis and Sylvie Lafrenière (2017), “Who Learns in a Business Game: An Objective Single Player Examination (Draft 6) Unpublished Manuscript. https://doi.org/10.1371/journal.pone.0137501 https://doi.org/10.1371/journal.pone.0137501 http://sloanreview.mit.edu/article/how-to-become-a-better-leader/#article-authors--outside http://sloanreview.mit.edu/article/how-to-become-a-better-leader http://sloanreview.mit.edu/article/how-to-become-a-better-leader