An Exploration of Overconfidence in Experiential Learning of Behavioral Skills among MBA Students Page 268 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 AN EXPLORATION OF OVERCONFIDENCE IN EXPERIENTIAL LEARNING OF BEHAVIORAL SKILLS AMONG MBA STUDENTS Robert C. Giambatista University of Scranton robert.giambatista@scranton.edu J. Duane Hoover Texas Tech University duane.hoover@ttu.edu ABSTRACT Several problems cited by instructors in organizational behavior (OB) (Burke & Moore, 2003) may be attributable in part to overconfidence among students. One question of interest to ABSEL scholars is the extent to which experiential learning environments interact with this phenomenon. While overconfidence is a well-known OB construct, its effect on acquisition of interpersonal behavioral skills in experiential learning settings is not well understood. In a study of MBA students exposed to an experiential behaviorally-based class featuring assessment centers, we found that overconfidence was a pervasive phenomenon, that it was an even larger phenomenon in the most interpersonally-oriented skills (leadership and teamwork), and we found that overconfidence was reduced between assessment centers and when disconfirming feedback was provided. Finally, we found some evidence that overconfident individuals performed more variably on a subsequent assessment center measurement. Implications for experiential learning theory and educational practice are discussed. INTRODUCTION AND THEORY All you need is ignorance and confidence, then success is sure. –Mark Twain Instructors of organizational behavior (OB) classes frequently lament that their students find the class to be obvious, common sense, or a waste of time (Burke & Moore, 2003). Specifically, Burke and Moore found that OB students reported more negative attitudes about their class than did accounting or computing students, demonstrated less interest in/attention to class, and perceived the class to be less relevant. Instructors also often attribute such attitudes to student apathy or lack of real world experience. However, such attitudes and their expression by business students can also be at least partially explained by overconfidence. ABSEL scholars have had a long term interest in how motivate students experientially (Gentry & McGinnis, 2008). In addition, concerns have been expressed about factors that can inhibit learning in experiential and simulation settings (Teach, 2010; Teach & Murff, 2009). Our research indicates that overconfidence could be one of the factors that should be considered as a potential inhibitor. The intent of this paper is to examine ways to address overconfidence as an inhibiting factor by examining the factors that contribute to its formulation and the ways it manifests in experiential learning settings focused on behavioral skill development. Educators’ classroom experiences with overconfidence may relate, in part, to the presence of narcissism. ABSEL scholars have commented on this phenomenon. For example, Markulis and Strang (2011) comment on mechanisms needed to engage Millennial generation students, a group characterized by higher levels of narcissism, in active learning. Twenge & Campbell (2010) make a compelling case that narcissism has been increasing generally for at least the last two decades. Moreover, there is evidence that narcissism is higher for business students than psychology students (Westerman, Bergman, Bergman & Daly, 2010). Long (2011) states that an appreciation for complexity should be part of an effective learning design, a sentiment echoed by Hoover (2011) who takes the position that complexity avoidance is one of the characteristics of narcissists and the Millennial generation. Pfeffer (2010) describes narcissism as “an inability to take the perspective of others, a dependence on others for affirmation and valuing oneself regardless of real achievements while seeking constant praise.” If we were to assume that overconfidence is one manifestation of narcissism, and given that such characteristics have been attributed to MBA graduates by both scholars and popular media/culture, it represents a significant challenge facing business and ABSEL educators. At its worst, it could represent a major challenge to both our graduate institutions as well as their finished products. Overconfidence is a phenomenon hardly confined to the classroom, of course. Managerial overconfidence and narcissism have been discussed in the popular press (Pfeffer, 2010, e.g.) and overconfidence is a well- mailto:robert.giambatista@scranton.edu mailto:duane.hoover@ttu.edu Page 269 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 researched construct in organizational behavior (Alicke & Govorun, 2005; Moore & Healy, 2008). One of the many troublesome manifestations of overconfidence is a general inability or unwillingness for an individual to perceive a need or desire for personal growth and development. For example, overconfidence has been described as the culprit behind individuals’ non-use of decision aids, even when such aids and their benefits to judgment are made plainly apparent and available (Arkes, Dawes, & Christiansen, 1986). Given the charge made by some that business schools should place more emphasis on improving behavioral skills (Rynes, Trank, Lawson & Ilies, 2003; Hoover & Giambatista, (2009); Hoover, Giambatista, Bommer & Sorenson, 2010) and the likelihood that individuals are overconfident regarding such skills, it follows that overconfident individuals might prove non- responsive to pedagogies designed to improve behavioral skills needed for effective management and leadership. Overconfidence has been studied in many disciplines; among those most relevant to management learning and education were findings that MBA students overestimate their future exam scores and number and starting salary of job offers (Armor & Taylor, 2002). In reviewing the Bernie Keys Library, the Academy of Management Learning & Education’s and the Journal of Management Education’s archives, we found little mention of overconfidence. We did find, however, that overconfidence was discussed by Henisz (2011) as a key “causal driver of the [recent financial] crisis.” Hoover et al. (2010:195) expressed that, “overconfidence can lead to insufficient emotional arousal and a failure to integrate a cognitive or behavioral experience into the learner’s schemas and repertoires.” In considering the perspective of classical Chinese thinkers, Rhee (2010: 268) concluded that overconfidence inhibits learning and performance by producing an “inadequate selection” of reference groups – in other words, a failure to look broadly enough and/or at the best perspectives to solve problems. Overconfidence has been found to be even more problematic in unfamiliar situations (Simon & Houghton, 2003), and it seems only a small leap to move from unfamiliar situations to those that seem familiar but are not quite what our intuition tells us they are. Many ABSEL scholars have commented over the years how experiential learning experiences and simulation settings put learners in unfamiliar territory, quite often by design. Another example of this phenomenon can be found in implicit theories of leadership (Meindl, Ehrlich, & Dukerich, 1985). Given the concerns expressed about overconfidence and hubris among executives (Kanter, 2005) and among MBA students (Pfeffer, 2010) it is odd that such little specific, formal treatment of overconfidence and its effect on learning and skill acquisition has been conducted. To begin remedying this deficiency, our exploratory study examines overconfidence with respect to the acquisition of behavioral skills in management. We consider the extent of overconfidence in an MBA classroom setting, whether it is robust across skill domains or more common in some areas than others, whether performance feedback on skills mitigates its prevalence, and examine overconfidence’s relationship to subsequent behavioral skill acquisition. THE PREVALENCE AND ROBUSTNESS OF OVERCONFIDENCE IN BEHAVIORAL SKILLS Overconfidence has been shown to be a substantial problem in human judgment (Moore & Healy, 2008) and was originally shown by comparing perceived likelihoods of events of phenomena or percentage confidence in having an answer correct versus actual data. A major consequence of overconfidence is its effect on the judgment and behavior of the actor. If overconfidence in behavioral managerial skills were similarly conceptualized as the gap between self-assessed perception of such skills and one’s actual demonstrated levels, it has been shown (Dunning and Kruger, 1999) that such inaccurate perception is likely to result in the belief that one’s performance level is already adequate or excellent. As a result, such individuals would have little motivational basis to attend to skill improvement. In a New York Times interview with Errol Morris (2010), Dunning summarized this perspective as, “If you’re incompetent, you can’t know you’re incompetent.” And more specifically, “When you’re incompetent, the skills you need to produce a right answer are exactly the skills you need to recognize what a right answer is. In logical reasoning, in parenting, in management, problem solving, the skills you use to produce the right answer are exactly the same skills you use to evaluate the answer.” Notice the recursive nature of the above dilemma – people who do not know they are lacking a skill cannot determine that they lack the skill. Since they have no perception of lacking the skill, they have little reason to try to acquire the skill. Thus, there is a twofold problem, one of actual skill deficiency, and the other of perception of and motivation to improve. If these phenomena prevail, the skill deficiency is likely to persist over time. Further, if the skill requires continuous improvement and/or maintenance, the skill level is likely to deteriorate over time. One of the reasons for overconfidence is biased feedback (Rosen & Tesser, 1970). Given that managerial behavioral skill areas are proximal to everyday social skills (communication, teamwork, leadership, decision-making, e.g.), it is fair to assume that we receive frequent feedback regarding these skills with friends, family, and acquaintances. Feedback is generally known to be biased, in part due to the “mum effect” associated with transferring negative feedback (Rosen & Tesser, 1970). This effect has been found in communicating unpleasant information – senders of such messages typically self-censor the negative aspects of the communication, leaving the receiver unaware of potential problems. Additionally, people generally fail to search for disconfirming evidence in their judgment Page 270 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 (Einhorn, 1980) and behavior. Thus, individuals would tend to persist more or less automatically in interpersonal behaviors that received positive feedback, thus reinforcing and further internalizing those behaviors, rather than challenging such behaviors and habits through self-inquiry in a proactive effort to better oneself through change. Pfeffer (2010) decried that college students in general lack the ability to engage in perspective-taking, seek constant praise, and are dependent on others for affirmation, characterizing but not citing a meta-analysis of narcissistic personality traits (presumably Twenge & Campbell, 2010). It can thus be assumed that young people are being socialized and acculturated in a manner that produces these characteristics, implying that parents, teachers, and other authority figures are over- complementing children and facilitating overconfidence through biased feedback. For example, Pfeffer (2010) states that if students did not arrive at business schools already narcissistic, participating in orientation activities extolling their virtues would soon make them so. Further exacerbating the problem, many authority figures are likely to themselves possess erroneous and oversimplified models of effective interpersonal skills. Consider, for example, implicit models of leadership (Meindl, Ehrlich & Dukerich, 1985), which argue that people portray leaders along simplified and stereotyped guidelines of dominant, “take charge” types. While we are not aware of implicit models of teamwork, we could easily infer that followers and team members are often stereotyped as the quiet, submissive yin to the yang of the stereotyped dominant implicit leader. Over years, biased feedback would tend to both reinforce these stereotypes and reinforce an unduly rudimentary and overly positive impression of an individual’s skill level in these and related areas of interpersonal effectiveness. Because overconfidence and narcissism are pervasive problems in human behavior, and because interpersonal skills are a centrally important manifestation of human behavior in organizations, we extend previous research in overconfidence to the area of interpersonal skills by arguing that: H1: An individual’s self-assessment of behavioral skills will be characterized by overconfidence such that self- assessed behavioral skills will be higher than objectively-assessed behavioral skills. Biased feedback can occur in both an individual and a social context. In other words, our mental scorekeeping of our judgmental accuracy is biased at the individual level regarding our perception of our own behavior, and is probably best known through the familiar self-serving bias (Miller & Ross, 1975). According to this bias, we tend to attribute success to our own personal qualities, and externalize failure to environmental and situational causes, which protects the ego but at the expense of creating a biased and overconfident self-concept. Similarly, the “better-than-average” effect (Alicke & Govorun, 2005) assumes that if a characteristic is innately desirable, the self probably scores better than average on that characteristic. Thus, some degree of overconfidence is probably based on our own biased self-perception. A second source of biased and overconfident self- perception can derive from feedback in social contexts, such as social interactions with friends, family, and colleagues. As described earlier, in these contexts, implicit models, mum effects, and over-complimenting are all likely to occur. Thus, in skill areas that are more personal in nature, we primarily obtain our biased feedback through biased self-perception. As skills become more interpersonal in nature, however, we would likely obtain biased feedback not only through biased self-perception, but increasingly through biased feedback received from interaction with others. While mum effects create a biased ‘box score’ of behavior, implicit models, when held by others, can create a second bias based on their superficial and invalid perceptions regarding interpersonal skills. Thus, as a skill becomes more interpersonal in nature, an even greater amount of biased feedback is experienced, further skewing one’s perception of, and confidence in, the skill domain. The most intensively interpersonal skills in the current study are leadership and teamwork, thus we contend: H2: Overconfidence will be more likely in team-oriented interpersonal behavioral skill areas such as teamwork and leadership versus relatively personal skill domains such as decision-making and planning/organizing. One remedy for overconfidence is to provide concrete performance feedback (Einhorn, 1980) that communicates skill deficiencies clearly and transparently to the individual. We rarely receive objective feedback from others on such skills, and even job performance appraisals containing criteria regarding attributes like teamwork, communication, leadership, and problem solving are often inflated and compressed due to perceptual biases of leniency and central tendency (Golman & Bhatia, 2012). A related problem, the aforementioned Dunning-Kruger effect, deals with what Donald Rumsfeld would have called “unknown unknowns” (Morris, 2010). When one lacks the self- awareness to understand that one lacks the skill (Kruger & Dunning, 1999), concrete and objective performance feedback can provide such self-awareness. Third party objectively scored assessment centers based on observational data, as utilized in this study, can be exceptionally helpful in this regard. Assessment centers generate carefully concocted scenarios that are deliberately designed to test behavioral skills important for managers, and assessed by expert raters, and provide feedback pointing to needed, specific improvements, thus navigating the skill-awareness dilemma latent in the Dunning-Kruger effect. Feedback showing a need for improvement can also be motivating in Page 271 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 the classic Skinnerian sense of providing negative reinforcement to improve performance and thus reduce the gap between self-assessment and demonstrated skill. At minimum, objective feedback provides an opportunity to re -calibrate one’s self-assessment, by providing a more well- informed ‘box score’ of one’s actual vs. expected performance (Einhorn, 1980), which can be helpful in managing overconfidence. Thus we argue: H3: Overconfidence prior to receiving behavioral skills feedback will be higher than overconfidence after receiving behavioral skills feedback. OVERCONFIDENCE AND THE ACQUISITION OF MANAGERIAL BEHAVIORAL SKILLS While it is certainly possible that negative feedback, or at least performance feedback that is lower than one’s self- assessment, might motivate positive performance, we should consider a couple of classic concepts in social psychology to entertain a more pessimistic possibility. For example, the self-serving bias (Miller & Ross, 1975) suggests that negative feedback is likely to be externalized rather than internalized. As a result, an individual may well dismiss the source of the unpleasant feedback, attributing it to something relatively benign like, “I had a bad day” to something more pernicious like, “These so- called ‘experts’ don’t know what they are doing.” Similarly, cognitive dissonance theory (Festinger, 1957) argues that dissonance raised by the gap between self- perception and performance feedback demands resolution, and one very likely possibility is that people will engage in denial regarding the performance feedback and its implications, as it conflicts with their self-concept as high- performing individuals. Narcissists are particularly subject to this characteristic (Carson, Butcher & Coleman, 1988; Millon, 1996), and while narcissism and overconfidence are different constructs, they clearly overlap. People who are overconfident may well have engaged in denial and externalization in the past, and thus are more likely to continue to employ them in processing such feedback than other individuals. Any substantive gap between self-assessment and performance feedback, however, puts individuals in a psychologically problematic place, relative to others not exposed to such a gap. Thus, overconfident individuals may be more prone to derogating or dismissing the disconfirming feedback’s implicit message regarding a need for behavioral skill improvement. In turn, this means they are less likely than others to attend to learning and developing these skills. H4: Overconfidence in behavioral skills will be associated with less improvement on those behavioral skills. One way to manage overconfidence is to confront it with disconfirming feedback (Kluger & DiNisi, 1996, conducted a meta-analysis on feedback interventions). In interpreting their meta-analytic results, Kluger and DiNisi (1996) found that feedback’s effectiveness varied greatly by several contextual variables that are not yet well understood. This variability suggests individuals process feedback in different ways depending on several factors, thus there is no guarantee that an individual will internalize such feedback and apply diligence to improve their skills. Consistent with our dissonance argument raised earlier, disconfirming feedback might also be simply dismissed to avoid unpleasant thoughts regarding one’s interpersonal skills. In sum, we believe that individual differences in how people manage disconfirming feedback will lead to very different behavioral choices. Consistent with the intent of feedback interventions, some individuals will accept the feedback and be motivated to eliminate its innate discomfort by working diligently to improve their performance. Consistent with principles of self-serving bias and dissonance theory, however, other individuals will dismiss and delegitimize the feedback. As a result, we believe that subsequent performance will become more varied among overconfident individuals who receive disconfirming feedback relative to other individuals. Those who take the feedback to heart should improve considerably, while those who derogate the feedback source should stagnate, seeing no need to try to improve. H5: Overconfidence in behavioral skills will be associated with greater variability in subsequently-assessed behavioral skills. METHODS AND RESULTS Our sample consisted of 171 subjects spread out over three years who were enrolled in an MBA program at a large southwestern public university. Students in this MBA program have an average of five years work experience, and are 27 years of age. Women and minorities comprise 44% and 10% of the student population, respectively. In this program, students were required to take a skills-based organizational behavior class. In this class, students began the semester with a 3-hour assessment center designed to provide a baseline of each student’s skills in the areas of leadership, decision-making, organizing and planning, communication, and teamwork. Each of these five skill areas then became the focus of a module. Within each module, a behaviorally-based pedagogy called whole person learning (WPL) was employed. The assessment center and pedagogy was modeled from that used by Hoover et al. (2010) and interested readers can find more details there. The pedagogy is designed to present a focused but short presentation on the cognitive basis relevant to the module and essential to understand the rationale behind crucial behaviors associated with each Page 272 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 skill. The majority of class time is spent in a variety of tasks such as fishbowl role play and vicarious observation, personal role play experience, extensive feedback and discussion of students’ various experiences with the role play task, and integration of the discussion. Towards the end of the semester, students again took part in a 3-hour assessment center to help gauge their behavioral skill level/ improvement after exposure to the pedagogy. The scenario for the second assessment center changed but the basic activities (group meetings and series of individually based decisions and communications) of this assessment center were similar to those of the first. Prior to the first administration of the assessment center, students were asked to assess themselves on the five dimensions. This self-assessment was compared to the actual results of the first assessment center to gauge overconfidence. VARIABLES AND ANALYSES Scores for the dimensions were obtained through assessment center results. The assessment center is a proprietary instrument called Iliad. More information about their services can be found at http:// www.collegiateassessment.com/assessment.php. Iliad employs professionally trained raters and asks them to view videotapes and paperwork collected from the administered assessment center and tabulate the presence, absence, and occasionally magnitude of behaviors associated with the five skill dimensions. From these data, a raw score for each dimension is compiled, and this raw score is compared against a large historical database to derive percentile performance data, much like GMATs, indicating the percentage of previous subjects who scored lower than the individual in question. This percentile performance comprised the pre- and post-test scores reported here. Iliad’s own internal validation studies have reported the inter-rater reliability for the composite of Iliad dimensions at =.71. We were not privy, however, to reliability data specific to the current study, as Iliad does not generally release these data. Similarly, Iliad keeps most of the specific rating behavioral criteria proprietary In addition to the behavioral skill assessment provided by the objective third party, students self-assessed their skills in each of the five dimensions prior to the pre-tests. Self-assessments were captured as percentiles in the same manner as the Iliad data to facilitate comparison. Overconfidence, then, represents the difference in percentiles between the self-rated skill level and the objectively-rated skill level. For example, an individual who believed him or herself to be in the 90th percentile in leadership skills and scored in the 60th percentile on the Iliad leadership dimension pre-test would receive a leadership overconfidence score of 30. We used t-tests for H1-H4, and also used multivariate regression to test H4. H5 employed Bartlett’s test for m ea n s. d 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 3 1 4 P re -t es t 1 L ea d er sh ip 4 5 . 0 8 2 9 . 1 6 2 D ec is io n -M ak in g 4 7 . 1 2 2 6 . 5 0 . 5 2 * * * 3 P la n n in g 5 0 . 5 2 3 0 . 3 8 . 2 8 * * * . 4 2 * * * 4 C o m m u n ic at io n 5 4 . 0 4 2 7 . 7 0 . 4 7 * * * . 5 3 * * * . 3 2 * * * 5 T ea m w o rk 4 7 . 2 0 2 8 . 4 2 . 3 5 * * * . 2 5 * * * . 2 5 * * * . 1 3 + P o st -t es t 6 L ea d er sh ip 6 3 . 2 1 2 5 . 7 6 . 2 9 * * * . 4 1 * * * . 2 7 * * * . 3 6 * * * . 1 8 * * 7 D ec is io n -M ak in g 6 6 . 5 3 2 5 . 1 0 . 0 9 . 2 6 * * * . 1 5 * . 3 6 * * * . 1 5 * . 3 6 * * * 8 P la n n in g 7 0 . 8 2 2 4 . 9 2 . 1 1 . 3 5 * * * . 4 0 * * * . 3 2 * * * . 1 1 . 3 8 * * * . 4 7 * * * 9 C o m m u n ic at io n 7 0 . 3 3 2 4 . 9 8 . 2 5 * * * . 4 1 * * * . 2 5 * * * . 4 7 * * * . 1 6 * . 5 5 * * * . 4 9 * * * . 5 0 * * * 1 0 T ea m w o rk 5 4 . 1 9 2 7 . 6 1 . 1 6 * . 2 0 * * . 1 5 * . 1 5 + . 2 0 * * . 2 1 * * . 1 7 * . 1 3 + . 2 5 * * O v er co n fi d en ce 1 1 L ea d er sh ip 2 5 . 1 4 3 0 . 2 6 - . 8 6 * * * - . 3 8 * * * - . 2 5 * * * - . 3 6 * * * - . 2 1 * * - . 2 1 * * - . 0 2 - . 0 5 - . 1 4 + - . 1 8 * 1 2 D ec is io n -M ak in g 2 2 . 6 9 2 8 . 2 8 - . 3 8 * * * - . 8 5 * * * - . 3 4 * * * - . 4 6 * * * - . 1 7 * - . 3 3 * * * - . 1 9 * * - . 3 0 * * - . 3 1 * * * - . 2 5 * * . 4 1 * * * 1 3 P la n n in g 2 2 . 2 4 3 3 . 7 1 - . 2 3 * * - . 3 4 * * * - . 8 8 * * * - . 2 5 * * * - . 2 2 * * - . 1 8 * - . 1 5 + - . 3 5 * * * - . 1 8 * - . 1 7 * . 3 0 * * * . 3 8 * * * 1 4 C o m m u n ic at io n 1 1 . 9 4 2 9 . 0 6 - . 3 0 * * * - . 3 8 * * * - . 2 7 * * * - . 8 2 * * * - . 0 1 - . 2 2 * - . 2 7 * * * - . 2 7 * * * - . 2 9 * * * - . 0 9 . 3 7 * * * . 4 7 * * * . 2 8 * * * 1 5 T ea m w o rk 2 6 . 0 4 3 0 . 5 2 - . 2 0 * * - . 1 3 + - . 2 3 * * - . 0 7 - . 8 8 * * * - . 1 2 - . 1 1 - . 0 9 - . 0 8 - . 1 7 * . 2 1 * * . 2 1 * * . 2 9 * * * . 0 9 T A B L E 1 D E S C R IP T IV E S A N D C O R R E L A T IO N S n = 1 7 1 . + i n d ic at es p < .1 0 . * i n d ic at es p < .0 5 . * * i n d ic at es p < .0 1 . * * * i n d ic at es p < .0 0 1 . http://www.collegiateassessment.com/assessment.php http://www.collegiateassessment.com/assessment.php Page 273 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 homogeneity of variance. For H1, we tested whether average overconfidence was significantly greater than zero. For H2, we tested whether average overconfidence in the most intensively interpersonal areas (leadership and teamwork) was significantly greater than the least interpersonal areas (decision-making and planning/ organizing). As Iliad’s communication dimension possesses both personal and interpersonal elements, we excluded it from a test of H2. For H3, we collected self- assessed skills just prior to the administration of the post- test for a subsample of our data. For these individuals, we compared pre-test overconfidence (self-assessed minus objectively rated) to post-test overconfidence, with H3 arguing that this difference would be significant. For H4, we conducted the t-test by using median splits on overconfidence, thus comparing post-test scores for those above vs. below median pre-test overconfidence for a given dimension. We also regressed post-test scores using pre- TABLE 2 RESULTS FOR ALL HYPOTHESES BY BEHAVIORAL SKILL DIMENSION Note: + indicates p<.10. ** indicates p<.01. *** indicates p<.001. H1-H4 each depict results of t-tests, while H5 depicts results of Bartlett’s tests for homogeneity of variance. 1. Data in this row represent overconfidence on the pretest - the difference in self-assessed skill level versus objectively rated skill level. Null hypothesis is that this difference is zero, research hypothesis is that the difference is greater than zero. n=171. 2. Data in these rows represent the difference in overconfidence between the interpersonal dimension specified in the row label (leadership or teamwork) and the dimension specified in the column label (decision making or planning). n=171. 3. Data in this row represent the difference in overconfidence between the pre-test and the post-test. Negative numbers indicate a decrease in overconfidence, consistent with H3. n=33. 4. Data in this row represent the difference in posttest scores between two subsamples generated by median splits on pre -test overconfidence. Negative numbers indicate lower posttest performance by those who were above the median on overconfidence for the specified dimension. n=171. 5. Data in this row represent the difference in variance of the posttest score on this dimension between two subsamples generated by median splits on pre-test overconfidence. Variance for the above-median overconfidence subsample are presented before comma, and below-median overconfidence after the comma. n=171. Leadership Decision Making Planning Communication Teamwork H1: Presence of overconfidence1 25.14 *** 22.69 *** 22.24 *** 11.94 *** 26.04 *** H2: Personal vs. interpersonal: leadership2 2.45 (p=.16) 2.89 (p=.16) H2: Personal vs. interpersonal: teamwork2 3.35 (p=.12) 3.79+ H3: Change in overconfidence over time3 -17.60 *** -20.32 ** -.88 -22.64 *** -28.32 *** H4: Posttest comparison of median split, more overconfident vs. less overconfident4 -9.69 ** -9.30 ** - 12.13 *** -16.06 *** -10.12 ** H5: Posttest variance comparison of median split, more overconfident vs. less overconfident (overconfident, not overconfident) 5 676.60, 611.52 725.58, 496.87+ 679.41, 496.79 (p=.15) 722.50, 400.54 ** 754.63, 726.28 Page 274 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 test scores as a key control, as well as controlling for instructor. For H5, we used the aforementioned median splits and calculated the posttest score variance for above vs. below median overconfidence subsamples. Bartlett’s test assumes a null hypothesis of equal variance across two samples, and generates a chi-square statistic to test the hypothesis. RESULTS Our descriptives and correlations are summarized in Table 1. Not surprisingly, and as hoped for as a consequence of the behavioral skills course, average scores on the skill dimensions increased on the second administration of the assessment center. Overconfidence in one skill area was associated with overconfidence in another skill area; these coefficients ranged from .09 to .47, with all but the former being significant at p<.01. This indicates that those who were overconfident in one skill dimension were more likely to be overconfident elsewhere, suggesting that overconfidence in behavioral skills manifested as a generalized phenomenon. Overconfidence was also negatively correlated with performance in the second round; these 25 coefficients were all negative, with 18 being significant at p<.10. This result is consistent with our H4. Table 2 contains results for each of our hypothesis tests. H1 hypothesized that mean overconfidence would be greater than zero, and our results strongly supported this hypothesis. Across the five behavioral dimensions and measured via t-tests, self-assessments were higher than objective skill ratings from a minimum of 11.94 percentile points (communication, p<.001) to a maximum of 26.04 (teamwork, p<.001). H2 hypothesized that mean overconfidence would be higher in the most intensive interpersonal dimensions (leadership and teamwork) than it would be in the least intensive interpersonal dimensions (decision-making and planning/organizing). We calculated four t-test results, one for each comparison. Each result was in the hypothesized direction and approached significance, but only one comparison (teamwork vs. planning/organizing difference = 3.79, p<.10) was significant. We also aggregated the leadership and teamwork overconfidence into one variable comprising two observations for each individual subject (n=342), and compared them to a similarly constructed variable for decision-making and planning, allowing for a larger sample size and comparing the more interpersonal areas vs. the less interpersonal areas in one t-test. This difference, 3.12, was significant, p=.05. Thus, H2 was weakly supported. H3 hypothesized that overconfidence would decrease from the pre-test administration to the post-test administration. Thirty-three of our 171 subjects completed a second skills self-assessment prior to the post-test, allowing for a test of this hypothesis. As can be noted from Table 2, each of the five changes in overconfidence reflected lower levels of overconfidence for the post-test, and four of these were significant at the .01 level. Planning/organizing was the only dimension which was not significant, otherwise the level of overconfidence decreased by 17.60 (leadership, p<.001) to 28.32 (teamwork, p<.001) percentile points. H3 was thus supported. In addition to these results being significant for four of the five dimensions, these changes were quite substantial given the ranges of pre-test overconfidence (11.94 to 26.04) across TABLE 3 TEST OF H4: OVERCONFIDENCE AND IMPROVEMENT – REGRESSION ANALYSES Note: + indicates p<.10. * indicates p<.05. ** indicates p<.01. *** indicates p<.001. Leadership Decision-Making Planning Communication Teamwork Instructor 1 -2.19 -15.71 ** -14.19 ** -4.01 -1.98 Instructor 2 -9.60 -16.13 ** -5.18 7.36 -10.72 Instructor 3 -5.34 15.22 * 3.80 14.58 * -7.74 Pretest .40 ** .32 ** .29 * .62 *** .23 Overconfidence .16 .06 -.03 .21 * .02 R2 .11 .23 .24 .31 .06 Model F 3.94 ** 9.82 *** 10.29 *** 14.56 *** 2.08+ n 171 171 171 171 171 Page 275 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 the dimensions. Subjects were much better calibrated for their post-test regarding their self-assessed skills for each of the dimensions except in the area of planning/ organizing. H4 was tested via t-tests and regression analyses. T- test results are depicted in Table 2, while regression results are depicted in Table 3. Our t-tests were developed by splitting each of the five overconfidence medians, and comparing the post-test skill level of the above-median subsample to the post-tests of the below-median subsample. As can be seen in Table 2, each of these results were significant at the .01 level, and ranged from -9.30 (p<.01, decision making) to -16.06 (p<.001, communication). Thus, students who were relatively overconfident on the pre-test tended to score lower on the post-test, supportive of H4. [Employing t-tests for this hypothesis, however, are limited because overconfidence is associated with poor pre- test performance (see Table 1 correlations between pre-test and overconfidence variables, each of which is negative and 22 of 25 are significant at the .05 level). Other things equal, we would expect lower pre-test scores to correlate with lower post-test scores (again, Table 1 shows that all 25 coefficients comparing pre-test scores with post-test scores are positive, with 22 of them significant at the .05 level). Thus, we should control for pre-test performance in order to best assess whether overconfidence in a given skill dimension predicts post-test performance. In Table 3, we show the results of this regression, where we also controlled for instructor. As can be seen, in no case was overconfidence negatively and significantly associated with post-test performance, thus H4 was not supported. Finally, H5 was tested using Bartlett’s test for homogeneity of variance. Recall that we argued individuals who were overconfident by definition received disconfirming feedback from the Iliad results, and that we expected some of these individuals to take the feedback to heart while others would dismiss it, leading to large differences in motivational response and thus relatively large post-test performance variability among overconfident individuals. As in the H4 t-tests, we employed median splits to generate sub-samples of relatively overconfident versus other individuals. Table 2 depicts the variance for each of these subsamples across the five dimensions. For decision making (726 vs. 497, p<.10), communication (722 vs. 401, p<.01), results were significant, and they approached significance for planning/ organizing (679 vs. 497, p=.15). Thus, H5 was weakly supported. DISCUSSION Our research revealed some interesting findings regarding overconfidence in the area of experientially taught behavioral skills, providing some empirical confirmation that might be intuitively expected, but also some more nuanced and perhaps even counter-intuitive findings. First, we found a high prevalence of overconfidence among the MBA students in our study regarding their behavioral skill levels. This replicates some of the findings of Westerman, et al (2010) as well as positions taken by Markulis, Murff and Strang (2011) and Hoover (2011). We found some evidence that overconfidence is relatively more prevalent in more interpersonally-intensive (i.e., team-oriented) skill domains such as leadership and teamwork versus less interpersonally intensive behavioral skills such as decision-making and planning/organizing. We found that overconfidence was mitigated in the second administration of the assessment center, thus the performance feedback from the first administration seemed to have the desired impact in the class that it was administered in but not necessarily consistent with some of the literature on overconfidence. Finally, we found some evidence, particularly in the less interpersonally intensive skill areas, that overconfident individuals were more variable in their post-test scores. Our research has important implications for theory and practice. First, the overconfidence literature often focuses on the psychology of judgment and ultimately derives from the heuristics and biases literature (Einhorn, 1980; Lichtenstein, Fishoff & Phillips, 1982). While many have discussed both the prominence and problematics of overconfidence and its related construct of narcissism among MBA students and business students (Pfeffer,2010; Westerman et al, 2010 ), this is the first study we know of to directly explore overconfidence in the behavioral components at the core of a graduate education class in organizational behavior. Given the stigma of OB in some quarters, and the popular perception that OB content is common sense fluff (Burke & Moore, 2003), it is not surprising that we would find overconfidence in a class like this. Second, our finding that overconfidence is somewhat more prominent in highly interpersonal skill areas may help us further refine theory-building and theory-testing in this area. As sources of biased feedback become more numerous, we believe that overconfidence becomes an even larger obstacle in experiential education and programs focused on personal development. Regarding practical implications for ABSEL educators, there are several important contributions this study suggests. For one, any who are not already cognizant of overconfidence’s prevalence in the MBA classroom would be well-advised to disabuse themselves of the notion and instead consider its implications to teaching, learning, and personal development programs. Second, the finding that overconfidence might be more problematic in areas like leadership and teamwork suggests that more attention to these topics and developing strategies for countering such overconfidence would be efforts well spent. For example, Burke and Sadler-Smith (2006) found that the use of multiple metaphors is helpful in reducing overconfidence among instructors. This finding certainly Page 276 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 seems applicable to students as well, for the strategy helps “develop a more complete picture and set of alternatives for viewing or solving problems and may, therefore, minimize overconfidence bias (p. 178).” Similarly, showing how popular and implicitly held beliefs and attitudes expected to produce positive outcomes can be unsuccessful may help deflate overconfidence, and tracking these back to self- identified characteristics might help individuals see how their self-concept and behavioral repertoire might not guarantee success, thereby making individuals more receptive to learning and development. The finding that disconfirming feedback reduced overconfidence, especially if the subject population is assumed to have narcissistic tendencies, is of particular importance and suggests that educators need to develop methods for confronting MBA students with gaps between their overconfident beliefs and attitudes about their skills and reality. However, we acknowledge that this is pedagogically dangerous; there is, for example, no guarantee that students are going to respond favorably to such feedback, particularly in light of increasing narcissism in society generally and business students in particular (Westerman et al., 2010). In fact, our direct experience in teaching this skills- based class over the three years of this study, confirms just such an outcome. In retrospect, as we contemplate our student evaluations received from this class, we can anecdotally observe a negative relationship between our student evaluation scores with both the amount of feedback given to students as well as the quality of that feedback (specificity, timeliness and behavioral skill relevance). Given the reciprocity effect in student evaluation of instruction (Clayson, Frost, & Sheffet, 2006), such outcomes could present potential dilemmas to instructors. At institutions where student evaluations are taken at face value in assessing teaching performance, instructors could suffer negative consequences. Thus, educators should not be naïve regarding the potential downside of providing specific, timely and behaviorally relevant feedback to students that is disconfirming of their inflated self-images. This is especially true where institutional support for impactful or transformative learning is weak. The lack of support for H4 is actually good news. We believed prior to the study that overconfident MBA students might generally operate along the self-serving bias by dismissing the disconfirming feedback and lagging in their skill development. We were quite pleased to find ourselves mistaken in this belief. Taken as a whole in our study, students did generally take the feedback somewhat to heart. Disconfirming feedback seemed to serve an important pedagogical and developmental purpose and, if anything, helped legitimize the skills being taught in this organizational behavior class. We did run several post-hoc regression analyses on subsamples to check the robustness of our conclusions. For example, it might be that students who scored extremely low on the pre-test were simply caught off guard, not yet engaged in the class (it was administered at the start of the semester), or had other characteristics that would tend to confound our results. Interestingly, in some of these subsamples, the overconfidence coefficient was negative and approached significance. We are not reporting these, but we do think it fair to state that the overconfidence variable does not seem very stable across different subgroupings of pre- and post- test performance, and thus believe there may be aspects of our data that require future analysis, particularly mediating and moderating variables. Our finding for H5 is also important. In addition to suggesting future research on possible mediators and moderators that might explain why overconfident individuals show more variability on subsequent learning and performance, it urges practitioners to consider the classroom not as a unified whole but as a collection of differing subsets of students presenting differing needs and challenges. If, as we suspect, there is a certain and non- trivial base rate of narcissism in the classroom, one admittedly pessimistic, and perhaps impractical, solution would be to somehow quarantine the most dedicated narcissists from the other students so that their attitude does not infect the others. If Greek mythology is accurate, there may be little else we can do for narcissists. Narcissus, after all, was so obsessed with his reflection in the water that he was unable to pry himself from it until he died. Such individuals may be prone to being impervious to disconfirming feedback and the need to attend to and internalize improvements in one’s interpersonal skills. A more proactive option, however, might be to try to confront these individuals more directly. The value of transformative learning could be strongly emphasized. Efforts could be made to help these individuals see learning as a process of continuous improvement, regardless of where their skill level might be, or where they perceive it to be. If narcissistic individuals crave power, one way to obtain power is to improve leadership and teamwork skills. Of course, there are many who believe that helping narcissists attain power and become more skilled in their intended self-serving manipulations may be inherently unethical, and we share some grave reservations ourselves. Our study has limitations, of course. Our sample size was limited, and we did not have sufficient data on important individual differences that might better explain both significant and non-significant findings. Another limitation is the finding for H4. Ultimately, using feedback to address overconfidence presents a potential confound – the positive learning and motivational effects of negative reinforcement would seem to be offset by overconfidence’s persistence. Future research should learn more about how individuals process feedback with respect to overconfidence to help tease out and inform what we believe are multiple, sometimes offsetting effects generating this non-finding. A good future study would gather self-assessment data decoupled to the extent possible from the pre-test, with no priming effects making overconfidence readily apparent. In the class from which Page 277 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 data for our study was obtained, students were instead notified of their overconfidence (these were called unconfirmed weaknesses) specifically as a feedback device to try to stimulate motivation and legitimacy for the course. In everyday life, of course, we rarely receive prominent disconfirming feedback (Einhorn, 1980), so a different study making self-assessment data and feedback effects less transparent might generate different results. Similarly, we can only offer untested theoretical speculations as to possible mechanisms for differential variability in post-test performance observed in H5. We would recommend future research first consider tests of variability in helping identify the imprint of not-yet- observed mediators and moderators, and that goes for constructs and processes far beyond the scope of this study. For example, in the current case, we would expect that individual differences such as narcissism might inform who is more likely to dismiss feedback. Identification of such potential mediators and moderators presents important classroom challenges and coping strategies for ABSEL educators and future ABSEL scholarly pursuits. REFERENCES Alicke, M.D., Govorun, O. 2005. The Better-Than-Average Effect. In M.D. Alicke, D. A. Dunning, J.I. Krueger (Eds.), The Self in Social Judgment. Studies in Self and Identity: 85-106. Psychology Press. Arkes, H. R., Dawes, R. M., & Christensen, C. 1986. Factors influencing the use of a decision rule in a probabilistic task. Organizational Behavior and Human Decision Processes, 37: 93–110. Armor, D.A., Taylor, S.E. 2002. When Predictions Fail: The Dilemma of Unrealistic Optimism. in T. Thomas; D. Griffin, D. Kahneman (Eds.). Heuristics and biases: The psychology of intuitive judgment. Cambridge, UK: Cambridge University Press. Burke, L.A. & Moore, J.E. 2003. A perennial dilemma in OB education: Engaging the traditional student. Academy of Management Learning & Education, 2: 37-52. Burke, L.A. & Sadler-Smith, E. 2006. Instructor intuition in the educational setting. Academy of Management Learning & Education, 5: 169-181. Carson, R.C., Butcher, J.N. & Coleman, J.C. 1988. Abnormal psychology and modern life (8th edition) Glenview, IL: Scott, Foresman and Company. Clayson, D.E., Frost, T.F., & Sheffet, M.J. 2006. Grades and the student evaluation of instruction: A test of the reciprocity effect. Academy of Management Learning & Education, 5: 52-65. Festinger, L. (1957). A theory of cognitive dissonance. Stanford, CA: Stanford University Press. Gentry, J.W. and McGinnis, L.P. (2008). “Thoughts on how to motivate students experientially”. Developments in Business Simulation and Experiential Learning, Volume 35, 73-77. Reprinted in the Bernie Keys Library [Available from http:// ABSEL.org] Golman, R. & Bhatia, S. (2012). Performance evaluation inflation and compression. Accounting, Organizations, and Society, 37: 534-543. Hoover, J.D., Giambatista, R.G., Sorenson, R.L. & Bommer, W.G. 2010. Assessing the effectiveness of whole person learning pedagogy in skill acquisition. Academy of Management Learning and Education, 9: 192-203. Hoover, J.D. & Giambatista, R.G. (2009), Why have we neglected vicarious experiential learning? Developments in Simulation and Experiential Learning, Vol. 36, Reprinted in the Bernie Keys Library. [Available from http://ABSEL.org] Hoover, J.D. (2011). Complexity Avoidance and Narcissism in Experiential Learning. Developments in Simulation and Experiential Learning, Vol. 38, Reprinted in the Bernie Keys Library. [Available from http://ABSEL.org] Henisz, W.J. 2011. Leveraging the Financial Crisis to Fulfill the Promise of Progressive Management. Academy of Management Learning & Education, 2011, 10: 298–321. Kanter, R.M. 2005. What Theories Do Audiences Want? Exploring the Demand Side. Academy of Management Learning & Education, 4: 93-95. Kluger, A.N., DeNisi, A. 1996. The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119: 254-284. Kruger, J., and Dunning,D. 1999. “Unskilled and Unaware of It: How Difficulties of Recognizing One’s Own Incompetence Lead to Inflated Self-assessments,” Journal of Personality and Social Psychology, 77: 1121-1134. Lichtenstein, S., Fischoff, B. & Phillips, L.D. 1982. Calibration of probabilities: The state of the art to 1980. In D. Kahneman, P. Slovic, & A. Tversky (Eds.) Judgment under uncertainty: Heuristics and biases: 306-334. Cambridge University Press. Long, K. (2011). Appreciating Complexity: The Chief of Staffing of the Army. Developments in Business Simulation and Experiential Learning, 38, Reprinted in the Bernie Keys Library. [Available from http:// ABSEL.org] Markulis, P.M., Murff, E. & Strang, D.R. (2011) Should college instructors change their teaching styles to meet the Millennial student? Developments in Simulation and Experiential Learning, Vol. 38, Reprinted in the Bernie Keys Library. [Available from http:// ABSEL.org] http://ABSEL.org http://ABSEL.org http://ABSEL.org http://ABSEL.org http://ABSEL.org http://ABSEL.org Page 278 - Developments in Business Simulation and Experiential Learning, volume 41, 2014 Meindl, J.R., Ehrlich, S.B., Dukerich, J.M. 1985. The romance of leadership. Administrative Science Quarterly, 30: 78-102. Miller, D. T., Ross, M. 1975. "Self-serving biases in the attribution of causality: Fact or fiction?" Psychological Bulletin 82: 213–225. Millon, T. 1996. Disorders of personality. DSM-IV and beyond (2nd edition). New York: Wiley- Interscience. Moore, D.A., & Healy, P.J. 2008. The trouble with overconfidence. Psychological Review, 115: 502–517 Morris, E. 2010. The Anosognosic’s Dilemma: Something’s Wrong but You’ll Never Know What It Is (Part 1). The New York Times. Retrieved from: http:// opinionator.blogs.nytimes.com/2010/06/20/the- anosognosics-dilemma-1/ Pfeffer, J. 2010. The narcissistic world of the MBA student. The Financial Times. Retrieved from: http:// www.ft.com/intl/cms/s/2/6470ac1a-e8f9-11df-a1b4- 00144feab49a.html#axzz1iPUVQCeN Rhee, M. 2010. The pursuit of shared wisdom in class: When classical Chinese thinkers meet James March. Academy of Management Learning & Education, 9: 258-279. Rosen, S., & Tesser, A. 1970. On reluctance to communicate undesirable information: The MUM effect. Sociometry, 33: 253-263. Rynes, S.L, Trank, C.Q., Lawson, A.M., & Ilies, R. 2003. Behavioral coursework in business education: Growing evidence of a legitimacy crisis. Academy of Management Learning & Education, 2: 269-283. Simon, M., & Houghton, S. M. 2003. The relationship between overconfidence and the introduction of risky products: Evidence from a field study. Academy of Management Journal,46: 139–149. Teach, R. (2010) Learning inhibitors in business simulations and games. Developments in Simulation and Experiential Learning, Vol. 37, Reprinted in the Bernie Keys Library. [Available from http:// ABSEL.org] Teach, R. and Murff, E. J. T. (2009), “Learning inhibitors in business simulations and games,” Developments in Simulation and Experiential Learning, Vol. 36 191- 197, Reprinted in the Bernie Keys Library. [Available from http://ABSEL.org] Twenge, J.M., Campbell, W.K (2010). The narcissism epidemic: living in the age of entitlement. New York, NY: Simon and Schuster. Westerman, J.W., Bergman, J.Z., Bergman, S.M., Daly, J.P. (2010). Are business schools creating narcissistic employees? An examination of business student narcissism. 2010 Academy of Management Annual Meeting Best Paper Proceedings. http://opinionator.blogs.nytimes.com/2010/06/20/the-anosognosics-dilemma-1/ http://opinionator.blogs.nytimes.com/2010/06/20/the-anosognosics-dilemma-1/ http://opinionator.blogs.nytimes.com/2010/06/20/the-anosognosics-dilemma-1/ http://www.ft.com/intl/cms/s/2/6470ac1a-e8f9-11df-a1b4-00144feab49a.html#axzz1iPUVQCeN http://www.ft.com/intl/cms/s/2/6470ac1a-e8f9-11df-a1b4-00144feab49a.html#axzz1iPUVQCeN http://www.ft.com/intl/cms/s/2/6470ac1a-e8f9-11df-a1b4-00144feab49a.html#axzz1iPUVQCeN http://ABSEL.org http://ABSEL.org http://ABSEL.org