On 360 Assessments in University Graduate Classes with Comparison to Corporate Data using a Business Simulation as the Experiential Pedagogy Page 186 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 ABSTRACT There is no doubt that receiving feedback, which is constructive and unbiased, is truly a gift that should be cherished by the recipient. The most common form of feedback comes from 360 assessments. While 360 assessments have been utilized for decades within corporate environments there has been rather limited application of these tools within a university setting, in particular, during a business simulation. To the best of our knowledge, this is one of the first times that the same 360 assessment has been used with university graduate students and corporate employees across a wide variety of industries. The experiential pedagogy we use is a business simulation. Our premise is that a business simulation can effectively, within reason, emulate a real world environment to help students identify areas of strengths and weaknesses via a 360, and act as vehicle to enhance strengths and correct weaknesses thus accelerating their development needs prior to full time employment. Our primary results indicate that graduate experientially oriented classes can help students identify and begin to mitigate development needs. Teammate data is found that supports the student’s perspectives of themselves and is reasonably consistent with what we have found in corporate applications of the same instrument across varied industries. Key Words: Business Simulation, 360 Assessment, Experiential Learning, Development Needs, Competencies, Soft and Hard Skills CHINESE PROVERB AS OUR DRIVER If you want one year of prosperity grow grain, If you want ten years grow trees, But if you want one hundred years of prosperity grow people INTRODUCTION There is no doubt that receiving feedback, which is constructive and unbiased, is truly a gift that should be cherished by the recipient. The most common form of feedback comes from 360 assessments. While 360 assessments have been utilized for decades within corporate environments there has been rather limited application of these tools within a university setting, in particular, during a business simulation. Ingols and Shapiro [3] provide an excellent summary of 360 utilization in academic business classes and Özgen et al [5] for an application in senior engineering programs. Taylor [8] provides a comprehensive discussion around the relevance of both self- and 360-assessment with the critical linkage to competencies and skills needed in organizational environments. Taylor [8] states “There has been a clarion call for management education to become more relevant to management and leadership practices, as well as for educators to assist students with real world applications of what they learned in the class room.” To the best of our knowledge, this is one of the first times that the same 360 assessment has been used with university graduate students and corporate employees across a wide variety of industries. The experiential pedagogy we use is a business simulation. Our purpose is not to enter the debate of the pro’s and con’s of 360s but to 1. help our students identify their development needs and seek validation of these from their teammates in an experiential team based business simulation, 2. compare the student and teammate data to corporate data across varied industries using the same 360 assessment. The corporate data allows for a direct comparison and students can see the types of skill sets and competencies that they will have to be prepared for in the corporate world, and 3. to share an instrument that others in the academic world may want to use in their classes using an experiential format, such as business simulations, that involve teams. Our premise is that a business simulation can effectively, within reason, emulate a real world environment to help students identify areas of strengths and weaknesses via a 360, and act as vehicle to enhance strengths and correct weaknesses thus accelerating their development needs prior to full time employment. Our primary results indicate that graduate experientially oriented classes can help students identify and begin to mitigate development needs. Teammate data is found that supports the student’s perspectives of themselves and is reasonably consistent with what we have found in corporate applications of the same instrument across varied industries. Tiwari et alia [9] found in academic business simulations that team dynamics emerged as the most critical dimension of overall learning. Of course, that fact ON 360 ASSESSMENTS IN UNIVERSITY GRADUATE CLASSES WITH COMPARISON TO CORPORATE DATA USING A BUSINESS SIMULATION AS THE EXPERIENTIAL PEDAGOGY Thomas E. Conine, Jr. Fairfield University, cone009@aol.com Barry Leskin Talent Management Consulting Page 187 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 that teams play the business simulation allows for the application of the 360. From the university perspective we need to ensure our students are well aware that jobs today are multi-faceted, that they will have numerous jobs in possibly varied locals and that being proficient operating in a team environment on-site and virtually is a given. Students also need to be aware they are really the only ones responsible for their careers. Given these constraints proper personal development is essential to remain competitive and a 360 can be a valuable tool in helping them prepare, but one of only many tools. 360S – PROS AND CONS Wikipedia [6] defines 360 Degree Feedback as “ In human resources or industrial psychology, 360- degree feedback, also known as multi-rater feedback, multi source feedback, or multi source assessment, is feedback that comes from members of an employee's immediate work circle. Most often, 360-degree feedback will include direct feedback from an employee's subordinates, peers (colleagues), and supervisor(s), as well as a self-evaluation.” While the corporate data we present is representative of the wide variety of feedback sources, our applications in our university graduate classes would be defined solely with peers (colleagues) as team mates and a self- evaluation both pre- and post- the simulation. It is well known that 360s, in corporate applications, have been, 1. Widely used (survey evidence has indicated that over 90% of the Fortune 500 has used 360s) 2. Primarily used for development with very few for performance appraisal 3. Viewed more positively than negatively 4. Shown to work best in organizations when the culture supports and values people development Certainly we are well aware of the negatives associated with office politics, either over or under leniency as a rater, and the defensive mechanisms as the receiver, subjectivity and biases. There has been some harsh criticism over the years. Most recently, Buckingham [2] provides an excellent treatise of potential negatives from a measurement perspective and they care needed in designing a 360 instrument. His key point is ensuring “… that your group of raters is perfectly representative sample of the competencies you are trying to measure.” Buckingham concludes there is value when the individual rates him/herself. Sigler and Rhee [7] make a compelling case, while drawing on corporate examples, of competency assessment from 360s style feedback as being more important to measuring student performance than traditional testing and final exams. TRI-LESKIN 360 QUESTIONS The assessment instrument we use is the TRI-Leskin 360. For our business simulations in either university and/or corporate settings we cull over 60 questions down to a subset of EXHIBIT 1 Page 188 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 12 that are business acumen focused on both soft and hard skill development. The questions can be found in Exhibit 1. Note the first 6 questions tend toward a hard skill focus for both macro- and micro-environments. The second 6 questions are tilted toward the soft skills. Naturally, any business leader needs to possess a balance of both. There is the constant debate about the ability of students can learn both or not. Two relevant academic studies that develop clusters of generic competencies and associated behaviors are Azevedo et alia [1] and Jackson and Chapman [4]. Both studies provide data, in particular [1], that appear to be reasonably correlated to the TRI-Leskin Assessment. These studies present data from Europe and Australia. Our TRI-Leskin assessment has been utilized in numerous simulations for companies in these regions of the world for decades. COMPARISON OF UNIVERSITY AND CORPORATE DATA The experiential exercise is one of TRI Corporations widely used business simulations that has been used in 14 of the Dow 30 as well as the graduate classes of one of the authors at Fairfield University. The breakdown of university and corporate data is given in Exhibit 2. In the university data the raters are the peers or team mates. We do have a wide variety of industrial structures to compare with. UNIVERSITY APPLICATION OF TRI-LESKIN ASSESSMENT Our TRI Corporation business simulations are experiential team based pedagogical tools designed to improve the overall business acumen of the participants. The simulations are cross- functional by design, financial metric and meeting commitments driven, and require balancing short- and long- term goals as well as soft and hard skills. There are numerous activities that link into the 12 assessment questions that allow team members to observe each other during the play of the simulation. These observations are in the privacy of the team’s breakout room. Many of our graduate participants (best estimate would be 30 to 40%) have also participated in corporate 360s (I do not keep data on specifics and, as such, the comments can be viewed as anecdotal). For that subset of students who have participated in a corporate 360 the general feeling is that the feedback validates what they hear in their business but they also feel the post discussion has less tension and much more of a true learning experience of their perceived strengths and weaknesses to identify developmental needs. The vast majority of the time the participants do not know each other coming into the business simulation teams. This helps reduce any bias that exist from working together in the past in other classes. Naturally, in a graduate program there are some exceptions. Occasionally, the participants come from the same company. By design when the simulations teams are formed we diversify on a various dimensions (gender, part vs full time, known corporate functions and academic backgrounds). Due to the nature of the simulation, and because the 360 process is discussed prior to coming to the course in pre -work, participants do sense accountability to each other in a non-threating environment. There is also no doubt the team mates are in a far better position to evaluate each other than we as instructors regarding the 360 questions. Our job as faculty is to provide the experiential aspects of the simulation, be available as a resource, and ensure that everyone understands the nature of giving and receiving feedback. We can only provide the environment. Prior to coming to a class the participants are asked to individually respond to the questions in Exhibit 1 as a self- assessment. The context is their life experiences, work background or whatever they feel can give the most insight into a self-observation. At the first team meeting, each member shares their personal assessment with their team mates. At no point do any of the other team member’s comment (even if someone states, they are exceptional in everything) on any pre- assessment. After the simulation has ended but prior to final debriefs each team member fills out Exhibit 1 on all team members, including doing a post assessment on themselves. The frame of reference is the simulation team activities were they have worked together for as much as 30 to 40 hours (typically over the course of a week in an intensive version of the class or over the course of 7 weeks on Friday evenings and Saturdays). Each team member takes approximately 30 minutes to fill out the form on all (including themselves). They must be able to sign their name. The team then meets privately for EXHIBIT 2 Page 189 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 approximately 60 to 90 minutes. Everyone gets back the signed feedback from everyone else (i.e., a team with 5 members, you would have your original pre-program assessment, your new one as well as 4 from the other team members). Each team members then has an opportunity to scan all of the feedback looking for commonalities and differences. The process then evolves a 10 to 15 minute discussion were each team member can openly ask anyone else how and why they came to the conclusions they did. These are totally private sessions and the only one whoever sees the feedback is the instructor (that generated the data within). Prior to the start of the private sessions there is a class discussion around giving and receiving feedback, along with the associated open-minded and listening that is needed. The participants are encouraged to review their feedback 1 and 6 months out. The only aspect of grading (10%) that is directly relevant to the 360 process within the simulation teams is that each participant must summarize their key learning’s about themselves from the process and what they will take forward in business and/or life in general. Specific questions they are encouraged to reply to, for the instructor, include, but is not limited to,  Were there any surprises (+ of -)?  How will the feedback help you in managing your career?  If you have participated in a corporate 360 (of course the questions may differ) is the current feedback aligned with that assessment?  Did you truly learn something about yourself?  What developmental needs have you identified to improve the probability of meeting your career goals? Naturally, we cannot confirm that every one of our students in the sample of 62 takes maximum advantage of the feedback they receive. It is very hard to know, for any one individual their true commitment to development, which is essential for the feedback to have lasting benefit or to ensure the student actively seeks out other developmental experiences to resolve problems identified. At a minimum we have provided an environment for the needs development to be identified but it truly is the student’s responsibility beyond that. Naturally, it would be tempting to correlate actual success in winning the simulation and scoring on the 360 but we purposely stay away from any measurement and the students are informed of such. We do have a numerical scaling factor for the various categories in Exhibit 1 that convert the results into a 1 to 100 scale. The scale is given by 51 to 60 for not observed, 61 to 70 for limited, 71 to 80 for proficient and 81 and above for exceptional. The not applicable, while it does occasionally generate a reply is not ranked. Exhibits 3 and 4 are respectively, the average values across all 12 questions. As expected, in Exhibit 3 we see a lower average value on the pre assessment. Exhibit 4 gives the comparison across a more detailed cut relative to the corporate data. In Exhibit 5 we breakdown the results by the 12 questions. The pre assessment shows 2 distinct areas. In all cases the pre assessment self-results are less than the post assessments with the greatest improvement seen in the first 6. Our purpose is not to debate statistical significance which can be elusive at best but to draw inference from trends and gaps analysis. As previously mentioned the first 6 are hard skill focus for both macro- and micro-environments. The second 6 questions are tilted toward the soft skills. It is not likely that the student (unless had work experience) would have encountered the need for knowledge and application of the first 6. Even with work experience many of the applications of the first 6 are often not seen until an employee gets to, at least, a manager level (some might argue a general manager level). The opportunity simply has not existed and other classes may have been very functionally based or EXHIBIT 3 Page 190 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 EXHIBIT 4 EXHIBIT 5 Page 191 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 none experiential. The post survey does reveal that, from their self-assessment, an increasing comfort level is felt with the first 6 and is reasonably aligned with the views of their team mates. The simulation has definite potential for applications in all areas related to the first 6. The second 6, tilted toward soft skills indicates a minimum of limited. Key development areas (below 70 on post scale) would include greater awareness of the global political, economic, social and technological change and impact on business, as well as financial acumen, relationship building, influence and conflict management skills, and developing a mindset around prudent risk taking. The conflict management skill enhancement as a development need is also supported by the team mates. In Exhibit 6 we add the consolidated corporate data for comparison. Of course, we recognize that care needs to be taken in drawing inference with the corporate data. There probably should be no surprise on the relative values of question 3, after all they do work in these industries were the students only spent a limited amount of time in the simulation. Question 7 also is not surprising and should be an important learning for our students that conflict management skills are highly valued in the corporate world. This can also be a signal not only as a development need but for the curriculums of graduate programs. Many programs we know devote that much time to experiential practice around conflict management (of course, it is routinely discussed in a lecture format). Question 12 should signal the criticality of prudent risk taking again a skill that is typically learned only on the job and by making mistakes. In the corporate world achieving growth and reward is much related to the ability to take prudent risks. Detailed breakdown by question within each industry setting as seen in Exhibit 2 is available upon request from the authors. Drawing inference from data is always very tricky and can have numerous interpretations. We are seeing from our preliminary analysis above that 360s can have a place in university experiential team- oriented classes and that there is some defined development needs that is commonplace across most students. Learning about these gaps as early as possible can give our students a competitive edge for their career development. SUMMARY If we can help in applications of 360s to your team-oriented experiential classes please contact us. We have learned that communication needs to be transparent, a process laid out from the beginning that all students understand, that privacy within the team of individual data, debate and dialogue is ensured, and that no grading is linked directly to any rating of the individual (only to what they learned about their individual development needs). In our opinion, the best use of 360s is for identifying development needs that will allow people to realize their fullest potential. The pro’s and con’s are well known and that debate will forever ensue, after all, we are dealing with people and their emotions as either raters or being evaluated. Understanding the giving and receiving of the feedback is crucial. This feedback can be one of the best gifts you ever receive. The 360 can identify your strengths and weaknesses from the perspective of others and yourself. You will need to expect gaps both positive and negative relative to your perspective. Naturally, the strengths and weaknesses can only be relative to the questions asked and may not totally represent all of them unless the survey is totally reflective of the leadership competencies you are looking at and nothing is perfect since the design itself is by people. EXHIBIT 6 Page 192 - Developments in Business Simulation and Experiential Learning, volume 42, 2015 We need to ensure our students are well aware that jobs today are multi-faceted, that they will have numerous jobs in possibly varied locals and that operating in a team environment in-site and virtually is a given. Students need to be aware they are really the only ones responsible for their careers. Given these constraints proper personal development is essential to remain competitive and a 360 can be a valuable tool in helping you prepare, but one of only many tools. ACKNOWLEDGEMENTS We would like to thank Lane Burkes for all quantitative results, Steve Mercer, Rich Avrut, Geraldine Mattia, Carla Haas, and our graduate students and corporate clients. REFERENCES Azevedo, A., Apfelthaler, G,. and Hurst, D., “Competency Development in Business Graduates: An Industry- driven Approach for examining the Alignments of Undergraduate Business Education with Industry Requirements” The International Journal of Management Education., pages 12 - 28, April 2012 Buckingham, M., “The Fatal Flaw with 360 Surveys” HBR Blog Network, 2011 Ingols, C., and Shapiro, M., “Concrete Steps for Assessing the ‘Soft Skills’ in an MBA Program” Journal of Management Education, pages 412 – 435, June 2014 Jackson, D., and Chapman, E., “Empirically Derived Competency Profiles for Australian Business Graduates and their implications for Industry and Business Schools” The International Journal of Management Education, pages 112 - 128, July 2012 Özgen, S., Alabart, J., and Medir, M., “Implementation of a 360-degree feedback assessment process for the development of the leadership competence of senior engineering students” Leadership and Management in Engineering, pages 65-75, April 2013 http://en.wikipedia.org/wiki/360-degree_feedback Sigler, T., and Rhee, K., “Unlocking Learning: Discovering the Keys to Effective Assessment” Journal of Management Education, pages 303 – 312, June 2014 Taylor, S., “Student Self-Assessment and Multisource Feedback Assessment: Exploring Benefits, Limitations, and Remedies” Journal of Management Education, pages 359 – 383, June 2014 Tiwari, S., Nafees, L., and Krishnan, O., “Simulation as a pedagogical tool: Measurement of impact on perceived effective learning” The International Journal of Management Education., pages 260 – 270, November 2014 *Thomas E. Conine, Jr. is professor of finance at Fairfield University in Connecticut. Dr. Conine is also president of TRI Corporation which specializes in corporate education, experiential leadership and simulation programs to help clients develop leaders, sharpen financial acuity, improve business acumen and enhance shareholder value. TRI Corporation clients have included 14 of the Dow 30. Dr. Conine has been an adjunct faculty at GE Crotonville since the early 80s. **Barry Leskin is president of Talent Management Consulting which offers services primarily focused on assessment and development of corporate employees, in particular, at the general manager level. Barry has been the CLO of Chevron/ Texaco and Ernst & Young. He served as Dept. Chair of Organization and Management at USC for nine years as well as Executive Director of their EMBA and HR professional track. His corporate clients have included Agilent, GE, IBM, HP, and BP. At GE he was actively involved in the early days of WorkOut at Crotonville.