Evaluation of the use of Assessment Centre methodology to enhance developmental planning, work placement outcomes and work readiness for postgraduate students – a pilot Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 45 Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot Sophie M. Keele 1 , Vanessa L. Sturre 2 , Kathryn von Treuer 3 , & Frances Feenstra 4 sophie.keele@deakin.edu.au 1, 2 & 3 School of Psychology, Deakin University 4 People Measures, South Melbourne Acknowledgements: We wish to thank Nick Carter for the provision of his expertise and knowledge in the development of the centre, Cameron Czerczyk who worked on this project as part of a work placement and Saville and Holdsworth Limited (SHL) for generously donating the Occupational Personality Questionnaire (OPQ) for use in the project. Abstract Work placements as a form of Work Integrated Learning (WIL) are widely recognised for the positive impact they have on improving the employability and work readiness of students. Assessment Centres (ACs) are typically used in corporate settings for recruitment, selection and more recently to provide developmental feedback to participants. The present project aimed to evaluate the application of AC methodology as a tool for measuring and subsequently enhancing professional competencies in a sample of postgraduate students in organisational psychology (n=15). A longitudinal design was utilised with numerous evaluation points from work placement stakeholders. This paper presents the first wave of findings. Students undertook a range of activities, including an in-tray exercise, a role play, a written report and a leaderless group discussion. Comprehensive feedback was provided to the students by Organisational Psychologists who also fulfil the role of Placement Co-ordinators. With the assistance of the Placement Co-ordinators, students prepared development plans relating to the competencies identified as requiring development. These development plans were to be addressed whilst on consecutive work placements. Initial perceptions gathered from students with regards to their participation in this initiative were very encouraging. Whilst based on a limited sample, the performance evaluations collected to date, as measured by behaviourally based ratings scales completed by the students themselves and their organisational supervisors, illustrate the positive effect of this methodology. The ongoing application of AC methodology as a process to enhance development planning, placement outcomes and work readiness for postgraduate students is discussed. Keywords: Assessment Centres, Development Centres, work readiness, Work Integrated Learning, work placements, graduate attributes, professional competencies. mailto:sophie.keele@deakin.edu.au Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 46 Introduction Work placements, fieldwork, industry-based learning, sandwich years, cooperative education, and internships are all methods universities use to equip students with knowledge of current workplace practices. Work placements, as they are referred to in the current paper, have become an integral part of many higher degree courses, under the umbrella initiative of Work Integrated Learning (WIL) (Smith, Brooks, Lichtenburg, McIlveen, Torjul and Tylor, 2009). WIL methods typically involve some form of interplay between workplace experience and formal learning as a part of a course of study in higher education. These methods are widely acknowledged as a superior vehicle for developing generic or professional skills and improving the employability and work readiness of students (Murakami, Murray, Sims and Chedzey, 2009; Patrick, Peach, Pocknee, Webb, Fletcher and Pretto, 2009). Work placements present themselves as an intensive, higher order form of WIL in so far as the student becomes engaged as an employee in a work setting for a specified period. Work placements may offer even more potential as a vehicle for developing generic or professional skills, employability and work readiness than more passive forms of WIL which typically involve observations, shadowing or case study analysis. Students can maximise their strengths, improve areas identified for development, and experience first hand some of the requirements of their chosen field within the confines of an authentic but well-monitored and rich learning environment. Employers often argue that whilst graduates are knowledgeable in their own discipline, they lack the communication, collaboration and other more generic professional skills required to make them productive without additional on the job training (Department of Education, Science and Training [DEST], 2007). Work placements provide the opportunity to address and accelerate this process. Students are often required to collaborate and communicate with colleagues as they complete work projects. They are also in a position to test their theoretical knowledge, putting it into action in an often fast-paced and complex working environment (Bates, Bates and Bates, 2007; Murakami, et al., 2009). Work placements are recognised for their impact on learning and employability by a range of stakeholders, including industry, government, universities and students (Bates, et al., 2007; Coll, et al., 2009). In a recent study examining the preparation for new professionals, Renn and Jessup-Anger (2008) found that students surveyed were virtually united in their agreement that practical experiences embedded in their course were relevant to a successful transition to life as a new professional. Such agreement with regards to the important role work placements play when transitioning out of the academic setting and into a professional setting full-time is echoed throughout the literature (Bates, et al., 2007; Crebert, Bates, Bell, Patrick and Cragnolini, 2004; Kadushin, 1992; Lefevre, 2005). Crebert, et al. (2004), in a study of graduates‟ perceptions relating to the relevant contributions made by the learning contexts of university, work placement and post-graduate employment to the development of their generic skills, found that graduates greatly valued the experience of learning in the workplace during placements. Graduates and employers believed that industry involvement during higher education was beneficial as it exposed students to real problems and gave them experience with the pressures commonly encountered as part of working life (Crebert, et al., 2004). Not surprisingly, the number of work placements within undergraduate and postgraduate courses is proliferating (Bates, et al., 2007). Higher student numbers and a greater number of courses incorporating some form of work placement have led to dramatic increases in the number of students participating in work placements. Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 47 As noted by Bates et al. (2007), universities are in a period of transition: work placement experience is receiving more emphasis in the higher education curriculum. This trend seems to be due to the increased demand for graduates who understand the role they play in shaping the organisations they enter and have the practical skills to contribute effectively in their roles (Bates, et al., 2007). However, effective contribution in the workplace requires more than mere possession of the necessary technical skills. It means engaging with the organisation and its goals, understanding the dynamics of the workplace, and taking up a job with an informed knowledge of all of its requirements. Effective contribution to the workplace also means applying a broad range of employability skills learned in many contexts and through a range of experiences (DEST, 2007). Work placements offer a key mechanism for learning and applying broad employability skills, such as those identified by the Australian Chamber of Commerce and Industry and the Business Council of Australia in 2002. Eight skills were identified as forming the Employability Skills Framework: communication skills, teamwork, problem solving, self-management, planning and organising, technology, life-long learning and initiative and enterprise skills (DEST, 2002). According to a report prepared for DEST, the Employability Skills Framework provided the initial starting point for discussion of employability skills in higher education in Australia. However, it is also noted that this framework must be considered in relation to the graduate attributes articulated by universities, and more specifically the competency frameworks offered by individual professions. Its analysis of graduate attributes from a number of Australian universities shows that employability skills, as outlined in the Employability Skills Framework, may reasonably be seen as a subset of graduate attributes (DEST, 2007). Therefore graduate attributes provide an appropriate starting point from which to further explore any future work on employability skills. Each discipline may also overlay its own professional competencies to these graduate attributes. It is the professional competencies of organisational psychology that superimpose the graduate attributes in the present study. However, regardless of the level of analyses, work placements offer varied and typically extensive opportunities for students to develop proficiency within the Employability Skills Framework as well as graduate attributes and more course specific competencies. Further to the increased emphasis on placement experience in the university curriculum, the federal government of Australia requested that universities become more accountable for the quality of such programs, requiring all stakeholders to address prescribed requirements (DEST, 2005). These requirements include the formalisation of responsibilities of stakeholders involved in work placements and the development of a more standardised approach towards work placements (DEST 2005). Additionally, the assessment of graduate attributes has been the subject of some discussion across education sectors and government bodies (DEEWR, 2008). Existing generic tools, such as the Graduate Skills Assessment (GSA) (Australian Council of Educational Research [ACER], 2005) and the Employability Skills Profiler (ESP) (Australian Chamber of Commerce and Industry [ACCI], 2007) have not been favoured by universities because the GSA is costly and considered too generic to be of value, and the ESP is generally considered to be more appropriate to non- professional job seekers (DEST, 2007). In an attempt to address the need to assess professional attributes, the application of AC methodology with a developmental focus was trialled in a sample of postgraduate students in Industrial/Organisational (I/O) Psychology. In addition to the assessment of professional attributes, the application of this methodology also addressed some of the prescribed requirements outlined, with regards to standardising the approach universities and students undertake both prior to and during work placements. Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 48 ACs have been widely utilised by organisations to identify and select appropriate job candidates for the past 50 years (Howard, 1997; Lievens, 2001). ACs employ a variety of techniques designed to allow participants to exhibit skills and abilities considered essential for successful job performance (Joiner, 1984). Commonly, participants undertake several job simulations or exercises designed to measure the competencies relevant to a given job. Exercises typically include role plays, in-tray exercises, leaderless group discussions, written reports and personality assessments. These are designed to measure competencies such as oral communication, written communication, problem solving and analysis, and interpersonal effectiveness. ACs are found to yield higher criterion related validity than other selection instruments and are well-regarded as a robust assessment technique (Howard, 1997; Robertson and Iles, 1988; Turnage and Muchinsky, 1984). More recently, single method selection tools, such as cognitive ability tests, personality assessment and structured interviews have been shown to provide more predictive power than previously thought, but when the objective of the assessment is diagnosis for individual development as was the case in the present research, AC methodology has advantages over tools that measure constructs which are not easily developable (e.g. cognitive ability) (Howard, 1997). The utilisation of AC methodology within the higher education setting is not new; however its application has been limited. The current paper provides an exploration of the value of AC methodology as a developmental tool in the work placement milieu. The developmental focus infers that the information gleaned during the assessment is used to identify strengths and development needs which are subsequently addressed. This was the exact application of the methodology in the present design. As mentioned, a number of researchers have applied AC methodology to the higher education setting, but typically the methodology has not been applied with relation to work placements and has not adopted a developmental focus (see Mullin, Shaffer and Grelle, 1991; Riggio, Mayes, and Schleicher, 2003). One published study undertaken in the United States, which did in fact use AC methodology as a developmental tool in a sample of graduate students in applied psychology, is that of Kottke and Shultz (1997). Six competencies were identified via job analyses, including written communication, oral communication, problem solving, organising, interpersonal, and organisational survival skills. Four exercises, namely a leaderless group discussion, oral presentation, an in-tray task and a role play were designed to measure the competencies. Written feedback was provided to students to use in career development planning. The present design aimed to elaborate on this innovative study and adapt it to an Australian-based course in I/O Psychology. AC methodology was used not only to assess a set of competencies prior to placements with a view to designing development plans to be actioned on placement, but also review and refine these plans after each subsequent placement. A personality assessment was also incorporated into the design and comprehensive one-on-one feedback was provided to participants. Furthermore, the present design incorporated a recommendation from Kottke and Shultz (1997), namely, running the centre in a contiguous block. Importantly, as noted by Kottke and Shultz (1997), whilst this research describes the application of AC methodology in the context of a Masters degree in I/O Psychology (MIOP), the methodology can be applied to a broad range of higher degree courses which include work placements as part of their curriculum. Education and health related disciplines present themselves as amenable to the AC process, but others such as information technology and environmental sciences are equally appropriate. The methodology outlined in this paper with regard to identifying and subsequently Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 49 measuring and developing the relevant competencies is flexible and can be adapted to diverse requirements by modifying the competencies and exercises to reflect current and future needs of a profession (Howard, 1997). Ultimately, it is employers who assess a graduate‟s employability and suitability for work when graduates apply for paid employment, but the methodology described provides early intervention, enabling students from a range of disciplines to address development needs with input from both university and organisational stakeholders based on an established, standardised process. ACs are probably best known for their use as a tool in the recruitment and selection of appropriate job candidates; however, over time their utility has become more expansive and, in line with the present design, many organisations use them as a developmental tool (Engelbrecht and Fischer, 1995; Howard, 1997; Iles, Roberston and Rout 1989; Woodruffe, 2000). As organisations attempt to address the need to develop staff, the application of AC methodology as a developmental tool has proliferated. In the main, the centres tend to be well received by participants, who typically react positively to the feedback provided and its developmental use (Engelbrecht and Fischer, 1995; Howard, 1997). Boehm (1985) specifies a number of conditions that should be met to ensure that the AC provides developmental value. Firstly, participants must be able to do something with the feedback they are provided. Secondly, the feedback must provide detailed behavioural examples and if possible relate these examples to job demands. Thirdly, the participants must be prepared to exert considerable effort and motivation to remedy areas identified as requiring further development, but must also be realistic in their expectations regarding possible change. Based on the demands placed on students enrolled in the masters program of I/O Psychology one can reasonably assume that they possess high levels of motivation, a willingness to learn and also a capacity to learn. The feedback sessions following the centre and the subsequent development plans were designed to be detailed and realistic, as well as task and competency specific. All conditions are therefore assumed to have been met. Based on a nexus of the conditions outlined by Boehm (1985), the literature promoting the use of AC methodology as a developmental tool (Engelbrecht and Fisher, 1995; Howard, 1997; Kottke and Schultz, 1997; Lorenzo, 1984; Robertson and Iles, 1988), and the vehicle provided by work placements to develop specific professional competencies, it was anticipated that participants would positively evaluate their experience of the centre known as the Postgraduate Development and Assessment Centre (PG-DAC). Positive evaluations were expected both after the feedback sessions, and after reviewing developmental plans subsequent to the first placement with regards to the contribution the process made in preparing for placements, providing a framework to address development needs and understanding the requirements of newly graduated Organisational Psychologists. Furthermore, whilst based on a small sample, it was of interest to examine whether the PG-DAC revealed sufficient levels of criterion-related validity, in terms of predicting the behaviourally anchored ratings completed by organisational supervisors at the completion of the first placement. In industry, ACs are often undertaken with small samples, particularly in the selection context, based on the assumption that they provide robust, dependable results, but they are rarely examined statistically. The opportunity to examine the outcomes from a well- established process with a small sample could not be ignored. Finally, it was also of interest to track students‟ self assessments and work placement supervisor assessments to detect whether improvements were made in an area specifically identified as a group development need. Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 50 Method The following section describes the process of identifying the professional competencies relevant to graduate employment as an Organisational Psychologist and the exercises designed to measure these competencies. The study was approved by the Human Research Ethics Committee of Deakin University. Participants All students (n=17) enrolled in the first year of the MIOP were invited to participate and 15 accepted this invitation. The majority of participants were female (93%). Materials and Procedure Competency Identification The competencies were determined through job analysis and competency modelling. Key stakeholders, industry bodies and the graduate attributes for Deakin University, Australia were consulted. Interviews were conducted with work Placement Co- ordinators (2), academics involved in lecturing, supervising placements and managing the course (3), and recent course graduates/practising psychologists (3). The industry bodies included the Australian Psychological Society and the College of Organisational Psychologists. These bodies provide a list of graduate attributes and competencies which were analysed to ensure that no critical behaviours were overlooked in the final specification of the professional competencies. Interviews ran for approximately one hour and utilised the Saville and Holdsworth Limited (SHL) universal competency cards (SHL Group plc, 2004) to identify the behaviours important for a newly graduated Organisational Psychologist. The competency cards were used to standardise the behaviours, ensuring that all stakeholders had similar interpretations of behaviours. Each card contains a list of behaviours that are relevant to a specific competency. Stakeholders were required to place each card on a four point scale ranging from critical or essential to not relevant. If stakeholders identified even one behaviour associated with the competency as critical, the competency was categorised as critical. In other words, not all factors listed on a card needed to be critical for the competency to be labelled as critical. Stakeholders were also required to provide workplace examples relating to the ratings and frequency of the behaviours. The professional competencies identified and the associated behavioural dimensions are presented in Table 1. These competencies are typical of those often found describing graduate attributes and managerial skills (Deakin, 2010; Howard, 1997) Masters of Industrial/Organisation Psychology Program The MIOP program at Deakin University is completed over two-years full-time and incorporates three main components, namely coursework, research, and work placements. Students are required to complete a total of 125 days of placement over two years in a minimum of three different organisations. The development and assessment centre was designed to be implemented prior to students undertaking work placements, with the specific aim of designing development plans to be actioned and monitored whilst on work placements. Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 51 Table 1: Professional Competencies and Behavioural Dimensions Competency Behavioural Dimensions Oral Communication Speaks clearly and fluently (using appropriate language and grammar) Expresses opinions, information and key points of an argument clearly when communicating with colleagues and clients Portrays credibility when discussing relevant information Articulates presentations with skill and confidence in all settings Responds positively and quickly to the needs of the audience and to their reactions and feedback Written Communication Avoids the unnecessary use of jargon or complicated language Writes in a structured, logical way Structures information to meet the needs and understanding of intended audience Explains separate thoughts or subjects in separate paragraphs Planning & Organising Sets clearly defined objectives Plans activities and projects in advance and takes account of possible changing circumstances Identifies and organises resources needed to accomplish tasks Meets deadlines Able to resolve time conflicts Consistently confirms plans and objectives with relevant parties Interpersonal Effectiveness Actively listens to all people, at all levels Consults others and communicates proactively when working in multidisciplinary teams Demonstrates an interest in and understanding of others Understands team dynamics and can adapt to different roles within a team Builds an effective network of contacts inside and outside the organisation Relates to people at all levels Deciding & Initiating Action Able to manage conflict resulting from change Makes specific recommendations in line with the organisations expectations, policies, procedures and intentions Seeks opportunities for organisational improvement Takes initiative and works under own direction when required Makes decisions under pressure Generates activity Problem Solving & Analysis Considers the practical issues related to implementing different solutions Considers all options/stakeholders/points of influence in determining and solving problems Makes decisions for the organisation using evidence based methods Produces workable solutions that meet the demands of the situation Demonstrates an understanding of how one issue may be part of a much larger system Look for causes of problems as well as identifying problems themselves Breaks information into component parts, patterns and relationships Probes for further information or greater understanding of the problem. Readily asks questions Makes rational judgements from the available information and analysis Organisational Alignment & Awareness Demonstrates an understanding of organisations and how they operate Works in a way to best advance business strategy within an organisation Understands the mindset of organisations and business in terms of bottom line goals Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 52 Exercise Design The Guidelines and Ethical Considerations for Assessment Center Operations were consulted in the development of the exercises (International Taskforce on Assessment Centre Guidelines, 2009). Four exercises were developed to measure the competencies outlined in Table 1. A personality assessment, the Occupational Personality Questionnaire (OPQ) (SHL Group, 2005), a measure of work preferences was also incorporated as an auxiliary tool for discussion in feedback sessions. Table 2 provides an overview of each exercise. The exercises were intended to provide a realistic simulation of applied problems, such that taken together they became „a day in the life of an Organisational Psychologist.‟ Table 2: Description of Assessment Centre Exercises Exercise Description In-tray TASK 1 Planning & Organising Assuming the role of an absent Senior Organisational Development Consultant, participants were asked to identify and group together associated items from the in-tray, decide on topic headings and prioritise the topics as high, medium or low priority, relating to both importance and urgency. Time given: 50 minutes. TASK 2 Decision Making Participants were required to make decisions about two separate issues selected from the in-tray. Time given: 15 minutes. TASK 3 Written Communication Participants were required to write a brief synopsis of the justifications, results and implications of a Leadership Development Program, based on the contents of the in-tray, for preparation of an article for a staff newsletter. Time given: 25 minutes Meeting & Presentation Role Play An assessor plays the role of a member of the Executive Committee concerned about a contentious issue plaguing the Executive. The participant was required to extract information from the Executive in a meeting and subsequently present a plan to the Executive to handle the issue of concern. Time given: 15 minutes preparation for meeting, 30 minutes preparation for presentation. 15 minutes presentation time including questions. Written Report Participants were required to prepare a report for the Executive Committee relating to an in-tray item. Time given: 60 minutes. Leaderless Group Discussion Participants formed a focus group as representatives for the College of Organisational Psychology. The group was asked to identify key learning and development needs for the first three years of a professional career and discuss methods to address these. Groups comprised up to 6 participants. No designated leader. The group was to arrive at a conclusion after discussion and produce a summary list of needs and delivery methods. Time given: 5 minutes preparation and 40 minutes discussion. Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 53 The link between the competencies assessed and the exercises undertaken is presented in Table 3. Table 3: Competency and Assessment Centre Exercise Links ACTIVITIES  COMPETENCIES  In Tray Meeting & Presentation Role Play Written Report Leaderless Group Discussion Oral Communication ** * Written Communication * ** Planning & Organising ** * Interpersonal Effectiveness * ** Deciding & Initiating Action * ** Problem Solving & Analysis ** * Organisational Alignment & Awareness * ** Note: asterisks indicate that the competency is being assessed within the specified exercise; ** indicates strong evidence, or a weighting of 60%; * indicates somewhat weaker evidence and a weighting of 40%. Assessors undertook two hours of training prior to conducting the PG-DAC. The operational approach described by Lievens (2001) in which all assessors do not rate all participants in every exercise was utilised. For each participant, one assessor was assigned to each exercise, ensuring that assessors did not measure the same competency twice (four assessors in total). Limiting the number of assessors can impose methodological limitations such as decreasing reliability (Howard, 1997); however, the aim was to develop a methodology that was resource effective and which could be applied in other higher education courses. Therefore, a centre that that was operational in nature and more easily replicated was implemented. Importantly, psychologists were used as assessors, which has been shown to improve criterion-related validities. When compared to managerial samples, psychologists have been found to show less difficulty in using AC constructs differentially (Gaugler, Rosenthal, Thornton and Bentson, 1987; Woodruffe, 2000). Furthermore, Woodruffe (2000) reports that the ratio of assessees to assessors does not have a significant effect on predictive validity. It should also be noted that two of the psychologist assessors were the current Placement Co-ordinators in the postgraduate course. The involvement of the co-ordinators was thought to be critical as they would play an ongoing role in the students‟ development through work placements throughout the course. The OPQ (SHL Group, 2005) was administered electronically and was sent to participants one week prior to conducting the job simulation exercises. Participants Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 54 took part in the job simulation exercises in one of three consecutive days. The in-tray was undertaken first, followed by an alternate schedule of either the meeting and presentation role play or the written report. These activities were subsequently rotated ensuring participants completed both exercises and any order effects were eliminated. The final activity undertaken was the leaderless group discussion. Data integration was then undertaken by assessors, whereby competency scores were aggregated across exercises based on pooled information from assessors. Placement Co-ordinators subsequently conducted feedback sessions of approximately two hours with participants individually. These sessions consisted of an explanation of the competencies assessed with relation to the activities undertaken, a thorough debrief of the individual‟s performance combined with the results of their personality profiles, and an exploration of discovered strengths and areas for development. The discussion culminated in the design of plans to target the development of relevant competencies. Plans were written on a template specifically for this purpose and one copy was kept by the Placement Co-ordinator, whilst a second was given to the student. With consideration of the feedback provided, students were asked to complete a self-assessment based on the competencies. This provided a baseline for further self-assessments collected at the end of each placement. Surveys were also distributed to evaluate the AC process. A review of the developmental plan was undertaken at the completion of the first placement as well as a second survey assessing the impact of the AC process. Organisational supervisors also provided assessments of the students‟ competencies, at the beginning and end of the work placement using a behaviourally-anchored rating scale. Results An initial exploration of the data was undertaken prior to investigating the specific hypotheses. A common criticism of ACs is that whilst they predict external criteria well, the evidence for the construct validity of the competencies is not so encouraging (Lievens, 2001; Thornton and Gibbons, 2009). Despite the small sample, an exploration of the data was undertaken to detect whether the pattern found in other studies could be identified. Correlations within the competencies, but between the exercises were calculated as an indicator of construct validity (or monotrait-hetero method correlations). As captured by Pearson coefficients, these were, 0.76** for Oral Communication, 0.34 for Written Communication, 0.25 for Planning and Organising, 0.52* for Interpersonal Effectiveness, 0.40 for Deciding and Initiating Action, -0.04 for Problem Solving and Analysis and 0.11 for Organisational Alignment and Awareness, (whereby ** indicates significance at the 0.01 level and * at the 0.05 level). Given the weak to moderate coefficients, particularly with regards to the latter two competencies, further analyses were warranted. As such, correlations within the exercises and between the competencies (monomethod-heterotrait correlations) were computed to examine the within exercise relationships, or the “exercise effect” (Woodruffe, 2000). Factor analytic techniques would be most appropriate and are planned following further implementation of the PG-DAC, but given the current sample size were not tenable in this instance. A series of bivariate correlations are presented for the in-tray, meeting and presentation role play, written report and leaderless group discussion in Tables 4, 5, 6 and 7 consecutively. Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 55 Table 4: Within Exercise Correlations for the In-tray In-tray- Written Communication In-tray – Planning & Organising In-tray – Deciding & Initiating Action In-tray – Problem Solving & Analysis In-tray- Written Communication In-tray – Planning & Organising 0.50 In-tray – Deciding & Initiating Action 0.53* 0.76** In-tray – Problem Solving & Analysis 0.59* 0.70** 0.85** ** Correlation is significant at the 0.01 level, * Correlation is significant at the 0.05 level. Table 5: Within Exercise Correlations for the Meeting & Presentation Role Play Meeting & Presentation Role Play – Oral Communication Meeting & Presentation Role Play - Interpersonal Effectiveness Meeting & Presentation Role Play - Deciding & Initiating Action Meeting & Presentation Role play - Organisational Alignment & Awareness Meeting & Presentation Role Play – Oral Communication Meeting & Presentation Role Play - Interpersonal Effectiveness 0.69** Meeting & Presentation Role Play - Deciding & Initiating Action 0.73** 0.64* Meeting & Presentation Role play - Organisational Alignment & Awareness 0.77** 0.63* 0.70** ** Correlation is significant at the 0.01 level, * Correlation is significant at the 0.05 level. Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 56 Table 6: Within Exercise Correlations for the Written Report Written Report – Written Communication Written Report – Problem Solving & Analysis Written Report - Organisational Alignment & Awareness Written Report – Written Communication Written Report – Problem Solving & Analysis 0.72** Written Report – Organisational Alignment & Awareness 0.78** 0.86** ** Correlation is significant at the 0.01 level, * Correlation is significant at the 0.05 level. Table 7: Within Exercise Correlations for the Leaderless Group Discussion Leaderless Group – Oral Communication Leaderless Group – Planning & Organising Leaderless Group - Interpersonal Effectiveness Leaderless Group – Oral Communication Leaderless Group – Planning & Organising 0.64* Leaderless Group - Interpersonal Effectiveness 0.79** 0.53* ** Correlation is significant at the 0.01 level, * Correlation is significant at the 0.05 level. Upon reviewing the coefficients, it is apparent that the correlations from within the exercises and between the competencies typically share more of the variance than the correlations within the competencies and between the exercises. In other words, most of the hetero-trait monomethod correlations were higher than the monotrait- hetero method correlations. It would appear that even in this small sample exercise effects have impacted the data. A number of remedial strategies are suggested to remove this effect, with varying degrees of success (Lance, 2008; Woodruffe, 2000). These will be detailed in the discussion section. Participant Evaluation Participants were surveyed shortly after the feedback sessions. Follow-up surveys were undertaken subsequent to the first placement and reviewing development plans. The means and standard deviations from the questions are presented in Table 8. Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 57 Table 8: Participant Evaluation Questions after feedback sessions Mean Standard Deviation 1. Overall response to being involved in PG-DAC 4.23 0.60 2. Please rate the relevance of the activities completed to your future work environment 4.54 0.52 3. How effective were the activities at providing evidence of the relevant competencies? 4.46 0.52 4. The PG-DAC assisted me in identifying placement opportunities which are most suitable for me 3.69 0.85 5. The PG-DAC helped me to identify specific areas which I need to focus attention on during my placements 4.46 0.52 Questions subsequent to first placement and reviewing development plans 6. How effective do you feel the PG-DAC process has been in helping you feel prepared for your first placement? 3.81 0.60 7. How effective do you feel the PG-DAC process has been in providing an opportunity to address your development needs while on placement? 4.15 0.42 8. How effective do you feel the PG-DAC process has been in providing an opportunity to maximise your strengths while on placement? 4.12 0.53 9. How effective do you feel the PG-DAC process has been in contributing to your overall work readiness (i.e. to start with a new employer in a role as an Organisational Psychologist or similar role?) 3.90 0.81 Note: For question 1, responses ranged from 1= extremely negative to 5 =extremely positive. For questions 2 and 3, responses ranged from 1 = not at all relevant to 5 = highly relevant. For questions 4 and 5, responses ranged from 1=not at all to 5=to a significant extent. For questions 6 to 9 responses ranged from 1 = not at all effective to 5 extremely effective. (n = 13 for all questions). As shown in Table 8, the ratings across most questions were very favourable. Participants generally felt that the PG-DAC had been helpful in preparing for placements, providing opportunities to address placement needs, maximising the impact of placements and contributing to their readiness to practise as an Organisational Psychologist. Consistent with the descriptive statistics presented, comments from students emphasised the overall value of participating in the centre. Students were asked to comment on their overall response to the PG-DAC, the activities and their overall relevance, and provide suggestions for improving the PG- DAC process. Participants employed terms such as “rewarding”, “useful” or synonyms of such terms. Interestingly, the majority of participants (61.5%) also found the experience challenging or intense. A couple of example comments are: “Overall rewarding - demanding day - rewarding feedback session,” “Experience very useful - quite intense on the day - very appreciative of the opportunity.” Suggestions for improvements related to the need for further communication prior the centre and the independent nature of the content of the final activity, which was not linked to the earlier exercises. Fatigue on the day itself was also mentioned. Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 58 Criterion-Related Validity Correlations were calculated using z-score transformations based on the weighted ratings of the competencies from assessors at the PG-DAC and the behaviourally- based ratings from organisational supervisors at the beginning and end of the first placement. Pearson correlations are presented in Table 9. Table 9: Correlations between PG-DAC Scores and Organisational Supervisor ratings Organisational Supervisor Z-score  PG- DAC Z-score OC PG- DAC Z-score WC PG- DAC Z-score PO PG- DAC Z-score IE PG- DAC Z-score DI PG- DAC Z-score PSA PG- DAC Z-score OAA Beginning of first placement 0.22 0.33 0.48 0.48 0.46 -0.06 0.31 End of first placement 0.24 0.26 0.35 0.50 0.30 -0.21 0.15 Note that abbreviations of the competencies have been used. OC–Oral Communication; W C–Written Communication; PO–Planning & Organising; IE–Interpersonal Effectiveness; DI–Deciding & Initiating Action; PSA– Problem Solving & Analysis; OAA–Organisational Alignment & Awareness. As shown in Table 9, correlations indicate moderate to relatively high predictive validity across the competencies, apart from Problem Solving and Analysis. The relationship with regards to Problem Solving and Analysis suggests that the lower the score from assessors on the PG-DAC, the better the rating the individual receives from organisational supervisors at the end of the placement, whilst the correlation at the beginning shows no relationship at all. Explanations for this outcome will be explored in the final section of this paper. As one would expect, the majority of coefficients revealed a stronger relationship at the beginning of the placement rather than the end. Development Needs Written Communication was identified as a development need for a number of the participants (n= 9), hence this competency was examined across time points using paired t-tests to detect whether improvements were made according to both self assessments and organisational supervisors assessments. Self assessments made prior to the first placement (M = 5.40, SD = 1.96) and at the end of the first placement (M = 6.90, SD = 1.29) showed a significant improvement over time, t (8) = 0-4.03, p = 0.00. Similarly, organisational supervisor assessments showed a significant improvement over time, with the mean at the beginning of the placement, (M= 6.67, SD = 1.22) significantly lower than the mean that the end (M = 7.89, SD = 1.17), t (8) = - 0.377, p = 0.01. Some improvement would be expected on all competencies over time; however the PG-DAC was instrumental in identifying specific areas as development needs requiring particular emphasis during work placements. Discussion The present project aimed to evaluate the application of AC methodology as a tool for measuring and subsequently enhancing professional competencies in a sample of postgraduate students in organisational psychology. Students completed a variety of job-relevant activities culminating in the design of comprehensive development plans Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 59 to be addressed during work placements. The results of this pilot investigation were very encouraging. Similar to many other studies using AC methodology in a development context, the PG-DAC received positive feedback from the participants involved (Boehm, 1985; Engelbrecht and Fisher, 1995; Howard, 1997; Iles, Robertson and Rout, 1989; Kottke and Shultz, 1997; Lorenzo, 1984). According to Howard (1997), participants tend to find AC exercises difficult, but believe that they assess relevant competencies and are useful for their development. The present study supports this general finding. Favourable feedback was received both prior and subsequent to the first work placement undertaken by students. In sum, students perceived the centre to be useful and effective in their pursuit of the competencies relevant to their employment as an Organisational Psychologist. Suggestions were made in terms of re-designing the leaderless group discussion and improving the communication designed to inform students prior to the centre. The content of the leaderless group discussion was based on a scenario that was unrelated to the other activities; however, it will be re-designed to better integrate with the other activities. Improvements are also planned for a more comprehensive communication strategy prior to the centre with the inclusion of additional communication mediums e.g. individual conversations with participants. In addition to the current feedback received from participants, it would be beneficial to follow up with participants after graduation to determine if they found the PG-DAC beneficial in supporting them to develop appropriate strategies during their transition to work. This is intended. Despite a small sample size, predictive validity from the PG-DAC is in line with most other research in this area. ACs have a long history of predicting external criteria well (Gaugler, et al., 1987; Robertson and Iles, 1988; Thornton & Gibbons, 2009). More recent meta-analyses have found rather modest coefficients (Hardison and Sackett, 2004; Hermelin, Lievens and Roberston, 2007) and the present study revealed a range of coefficients from modest to relatively high. The results are promising, but without a larger sample and further research it is not possible to make conclusive remarks. Again, whilst the sample is limited, the pattern of results with regards to internal validity, and in particular the exercise effect, reflects that of prior research (Thornton and Gibbons, 2009; Woodruffe, 2000). The logical expectation is that the correlations within the competencies and between the exercises would be greater than the correlations within the exercises and between the competencies; however, the opposite is commonly found (Woodruffe, 2000) and this is true of the current data. Remedies, typically in the form of improvements in assessor training including increasing the length of assessor training, limiting competencies, and listing key behaviours have been suggested (Lievens, 2002; Schleicher, Day, Mayes and Riggio, 2002). These strategies have been found to reduce exercise effects (Lievens, 2002; Schleicher, et al., 2002); however, most of these were utilised in the present design. Nevertheless, improvements are planned for assessor training. The competencies and how the exercises measure them will be explained more thoroughly to assessors to ensure a common definition and a better understanding of the differences between competencies. As noted by Woodruffe (2000), if the competencies are not clearly separated in their definition, it is unlikely that there will be differentiation between them in the ratings. The key behaviours relating to the competencies will also be reviewed prior to further implementation of the PG-DAC. This is likely to be particularly important for the competencies of Problem Solving and Analysis and Organisational Alignment and Awareness, which revealed low correlations within the competencies and between the exercises (monotrait- Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 60 heteromethod). However, it is acknowledged that whilst remedial strategies lead to some improvement, they rarely succeed in producing a clear pattern of competency effects (Thornton and Gibbons, 2009). More recently, research suggests the treatment of exercise effects as measurement error is not warranted for ACs (Lance, 2008). Lievens (2002) suggests that exercise effects represent real variation in performance across exercises, whilst Lance (2008) concludes that candidate behaviour is situationally-specific, rather than cross-situationally consistent. Indeed it may be the case that the exercises, particularly those within the competencies of Problem Solving and Analysis and Organisational Alignment and Awareness reflect quite different situations. On the face of it, a role play and written report which are used to measure Organisational Alignment and Awareness appear dramatically different. Nevertheless, improvements have been planned. Finally, it would be of interest to examine data following future implementation using exploratory and confirmatory factor analytic techniques to detect whether an exercise factor does in fact exist. An unexpected outcome was found with regards to the competency of Problem Solving and Analysis. This competency revealed poor correlations within the competency and between the exercises, as well as poor criterion-related validity. It may be that the understanding of this competency was not consistent across the PG- DAC exercises and the work placement. PG-DAC assessors and organisational supervisors may not have had a consistent view of what this competency and its associated behaviours mean. Whilst the in-tray and written report were used to assess this competency, it seems unlikely that organisational supervisors would incorporate written reports in their assessment of Problem Solving and Analysis during a student placement. It seems more likely that they would consider the students‟ general approach to assigned tasks when assessing performance. The link between Problem Solving and Analysis and written reports may need to be more explicitly stated. The planned review of the key behaviours associated with the competencies could improve this outcome. It is also worth noting that this competency may be more easily measured with a cognitive ability test rather than a set of behaviours. Future examination of this competency following further implementation of the PG-DAC is required to provide more light on this issue. As mentioned, the present pilot study found that participants generally indicated that the experience was effective in providing an opportunity to address development needs, maximise opportunities, and contributed to their overall work readiness. An improvement was detected with regards Written Communication. This competency was identified as a development need for a number of participants and analyses show an improvement in this competency for the relevant participants based on self assessments and the less subjective organisational assessments completed by organisational supervisors. Whilst many factors may be contributing to this improvement, the PG-DAC provided the mechanism to identify and measure the competencies and in particular the need to develop the competency of Written Communication, which appears to have been addressed in the context of the work placement. This points to the importance and effectiveness of work placements as a vehicle for developing attributes required by employers. As mentioned, work placements are widely acknowledged as a superior method for developing generic or professional skills and improving the employability and work readiness of students (Patrick, et al., 2009; Smith, et al., 2009). Students participating in the development and assessment centre process are likely to have an advantage over others when it comes to addressing and accelerating the acquisition of desired skills. Students receive critical feedback at an early stage of their careers and can take targeted Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 61 remedial action well before entering paid employment in their chosen field. In sum, the rigour and comprehensive techniques offered by the AC methodology enables students to focus on and improve areas identified for development as well as maximising strengths. Assessing employability skills or graduate attributes has been the subject of some discussion within education sectors and government bodies. It is thought that organisational supervisors are in a unique position to assess and provide feedback on a student‟s employability skills (DEST, 2007). The application of AC methodology is likely to bring more exacting standards to this process. By employing AC methodology and providing organisational supervisors with behaviourally-anchored rating scales on relevant competencies, they are likely to be in a better position to assess desired professional skills. As mentioned, existing generic assessment tools such as the GSA (ACER, 2005) and the ESP (ACCI, 2007) are not favourably perceived. AC methodology appears to address a need that is currently unmet, as users can modify the competencies to reflect the current and future needs of their graduates. It must be acknowledged that AC methods are resource intensive, but given that the expertise to design and implement them exists within most higher education settings, cost savings are likely. Additionally, if the centres are designed to be operational as in the current design, costs can be further contained. Unfortunately, this can impose important methodological limitations. Lievens (2001) found that increasing the number of assessors scoring an exercise had a greater impact on the reliability of the centre than increasing the number of exercises. However, given the economic constraints facing most educational institutions, this is likely to become a trade-off with other more feasible strategies, such as using psychologists as assessors. Interestingly, Woodruffe (2000) recently reported that the ratio of assessors to assessees does not have a significant effect on predictive validity and if psychologists are used as assessors, the centre appears to not only be more predictive but also more construct valid than if psychologists are not used as assessors. A number of other researchers also support the use of psychologists as assessors (Gaugler, et al., 1987; Lievens, 2001; Sagie and Magnezy, 1997). It is important to note that the present design formed a pilot study and was undertaken with a limited sample. Future implementation is planned with larger samples, enabling a more comprehensive analysis of the methodology. Nevertheless, the process was successfully implemented with positive feedback and promising initial statistical findings albeit with a small predominantly female sample. Clearly it cannot be said that the results will generalise to other disciplines within the higher education setting or even to a larger body of organisational psychology students, but the study provides an innovative approach and an imperative to trial it within other disciplines. The application of the methodology appears to provide a much needed strategy for not only assessing professional competencies in a range of higher education courses, but also providing a standardised way forward for universities with regards to the preparation, planning and ongoing development of professional competencies in the context of work placements. Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 62 References Australian Chamber of Commerce and Industry. (2007). Skills for a Nation: A Blueprint for Improving Education and Training 2007-2017. Canberra: ACCI Retrieved 1 December 2010 from http://www.acci.asn.au/SkillsBlueprintMain.htm Australian Council of Educational Research (2005). Graduate Skills Assessment. Melbourne, Australia. Retrieved 1 December 2010 from http://www.acer.edu.au/tests/university/gsa/intro.html Bates, A., Bates, M., & Bates L. (2007). Preparing students for the professional workplace: who has responsibility for what? Asia-Pacific Journal of Co- operative Education, 8(2), 121-129. Boehm, V.R. (1985). Using assessment centre for management development – Five applications. Journal of Management Development, 4(4), 40-53. Coll, R.K., Eames, C., Paku, L., Lay, M., Ayling, D., Hodges, D., Ram, S., Bhat, R., Fleming, J., Ferkins, L., Wiersma, C., & Martin, A. (2009). An exploration of the pedagogies employed to integrate knowledge in work-integrated learning in New Zealand higher education institutions. Wellington, New Zealand: Crown. Crebert, G., Bates, M., Bell, G., Patrick, C., & Cragnolini, V. (2004). Developing generic skills at university, during work placement and in employment: graduates‟ perceptions. Higher Education Research and Development, 23(2), 147-165. Deakin (2010). Graduate Attributes at Deakin, Higher education courses - operational policy, Deakin University. Retrieved March 2010 from http://www.deakin.edu.au/itl/pd/tl-modules/curriculum/grad-attrib/grad-attr- 02.php Department of Education, Employment & Workplace Relations (2008). Australian Core Skills Framework, Canberra, ACT: Commonwealth of Australia. Department of Education, Science & Training (2002). Employability skills for the future. Prepared by the Australian Chamber of Commerce and Industry and the Business Council of Australia for the Department of Education, Science & Training. Canberra, ACT: Commonwealth of Australia. Department of Education, Science & Training (2005). Administrative information for providers. Canberra, ACT: Commonwealth of Australia. Department of Education, Science & Training (2007). Graduate Employability Skills. Prepared for the Business, Industry and Higher Education Collaboration Council. Canberra, ACT: Commonwealth of Australia. Engelbrecht, A.S., & Fisher, A.H. (1995). The Managerial Performance Implications of a Developmental Assessment Center Process. Human Relations, 48(4), 387-404. http://www.acci.asn.au/SkillsBlueprintMain.htm http://www.acer.edu.au/tests/university/gsa/intro.html http://www.deakin.edu.au/itl/pd/tl-modules/curriculum/grad-attrib/grad-attr-02.php http://www.deakin.edu.au/itl/pd/tl-modules/curriculum/grad-attrib/grad-attr-02.php Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 63 Gaugler. B.B., Rosenthal, D.B., Thornton, G.C. III., Bentson, C. (1987). Meta- analyses of assessment centre validity. Journal of Applied Psychology, 72(3), 493-511. Hardison, C.M., & Sackett, P.R. (2004). Assessment center criterion-related validity: A meta-analytic update. Paper presented at the 19 th annual conference of the Society for Industrial and Organizational Psychology, Chicago, IL. Hermelin, E., Lievens, F., & Roberston, I.T. (2007). The validity of assessment centres for the prediction of supervisory performance. International Journal of Selection and Assessment, 15(4), 405-411. Howard, A. (1997). A Reassessment of Assessment Centers: Challenges for the 21 st Century. Journal of Social Behavior and Personality, 12(1997), 13-52. Iles, P, Roberston I., & Rout, U. (1989). Assessment-based Development Centres. Journal of Managerial Psychology, 4(3), 11-16. International Taskforce on Assessment Center Guidelines (2009). Guidelines and ethical considerations for assessment center operations. International Journal of Selection and Assessment, 17(3), 243 – 253. Joiner, D,A. (1984). Assessment Centres in the public sector: A practical approach. Public Personnel Management Journal, 13(1984), 435-450. Kadushin, A. (1992). Supervision in Social Work. New York, Columbia University Press. Kottke, J.L., & Shultz, K.S. (1997). Using an assessment centre as a developmental tool for graduate students: A demonstration. Journal of Social Behavior and Personality, 12, 289-302. Lance C.E. (2008). Why Assessment Centers Do Not Work the Way They Are Supposed To. Industrial and Organizational Psychology, 1(1), 84-97. Lefevre, M. (2005). Facilitating practice learning and assessment. The influence of relationship. Social Work Education, 24(5), 565-583. Lievens, F. (2001). Assessors and use of assessment centre dimensions: a fresh look at a troubling issue. Journal of Organizational Behaviour, 22(2001), 203- 221. Lievens, F. (2002) Trying to understand the different pieces of the construct validity puzzle of assessment centers: An examination of assessor and assessee effects. Journal of Applied Psychology, 87(4). 675-686. Lorenzo, R.V. (1984). Effects of assessorship on managers‟ proficiency in acquiring, evaluating, and communicating information about people. Personnel Psychology, 37(4), 617-634. Mullin, R, G., Shaffer, P.L., & Grelle, M.J. (1991). A study of the assessment center method of teaching basic management skills. In J.D. Bigelow (Ed)., Managerial skills: Explorations in practical knowledge (pp 116 – 153). Newbury Park, CA: Sage. Keele, S.M., Sturre, V.L., von Treuer, K., and Feenstra, F. (2010). Evaluation of the use of Assessment Centre methodology to enhance development planning, work placement outcomes and work readiness for postgraduate students – a pilot. Journal of Teaching and Learning for Graduate Employability, 1(1), 45–64. 64 Murakami, K., Murray, L., Sims, D. & Chedzey K. (2009). Learning on work placement: The narrative development of social competence. Journal of Adult Development, 16(1), 12-24. Patrick, C., Peach, D., Pocknee, C., Webb, F., Fletcher, M., & Pretto, G. (2009). The WIL Report: Work Integrated Learning - A national scoping study. Prepared for the Australian Learning and Teaching Council. Brisbane, Qld: Queensland University of Technology. Renn, K.A., & Jessup-Anger E.R. (2008). Preparing New Professionals: Lessons for Graduate Preparation Programs from the National Study of New Professionals in Student Affairs. Journal of College Student Development, 49(4), 319-335. Riggio, R.E., Mayes, B.T. & Schleicher, D.J. (2003). Using Assessment Centers Methods for Measuring Undergraduate Business Student Outcomes. Journal of Management Inquiry, 12(1), 68 – 78. Robertson I.T & Iles, P.A. (1988). Approaches to Managerial Selection. In Cooper, C.L. and Robertson I.T. (Eds.), International Review of Industrial Organisational Psychology, Chichester: John Wiley. Sagie, A., & Magnezy, R. (1997). Assessor type, number of distinguishable dimension categories and assessment centre construct validity. Journal of Occupational and Organizational Psychology, 70(1997), 103-108. Schleicher, D.J., Day, D.V., Mayes, B.T., & Riggio, R.E. (2002). A new frame for frame of reference training: Enhancing construct validity of assessment centers. Journal of Applied Psychology, 87(4), 735-746. SHL Group (2005). OPQ32 Technical Manual. Thames Ditton, England: SHL Group plc. SHL Group plc (2004). Competency Profiler, Version 4.0, Thames Ditton, England: SHL Group plc. Smith, M., Brooks, S., Lichtenberg, A., McIlveen, P., Torjul, P.& Tylor, J. (2009). Career Development Learning: Maximising the contribution of work integrated learning to the student experience. Final project report. National Association of Graduate Careers Advisory Services. N.S.W.: University of Wollongong. Careers Central. Academic Services Division. Thornton, G.C., & Gibbons, A.M., (2009). Validity of assessment centres for personnel selection. Human Resource Management Review, 19(3). 169-187. Turnage, J.J., & Muchinsky, P.M. (1984). A comparison of the predictive validity of assessment center evaluations versus traditional measures in forecasting supervisory job performance: Interpretive implications of criterion distortion for the assessment paradigm. Journal of Applied Psychology. 69(4), 595-602. Woodruffe, C., (2000). Development and assessment centres: identifying and developing competence (3 rd Edition). London: Human Assets Ltd.