Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 152 Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award Toni Wright1 and Sarah Jeffries-Watts2 T.Wright@newman.ac.uk ; S.Jeffries-Watts@bham.ac.uk 1 Newman University, 2 University of Birmingham Abstract University employability awards, in the UK particularly, aim to assist students to develop career related skills and attributes and thus increase their potential to achieve graduate level employment. Self-report quantitative and qualitative data were collected at intervals via questionnaires, interviews and focus groups from two cohorts (N = 212) of a well-established career development and employability award at a large civic UK university. Findings indicated increases in confidence and aspiration, and in the ability to articulate and apply skills and abilities; also that the award may convey similar benefits to work experience. Survey data from award completers indicated that they had changed their career related behaviour, and students who have completed the award show a consistent small increase in their level of graduate employment when compared to the graduate employment figure for the university as a whole in the UK university destinations data. Employability related values, attitudes and behaviour may all change as a result of award experience. Reported changes implied a sense of improved resourcefulness. It is hypothesised that the award may enhance student employability somewhat via development of psychosocial resources, producing a shift in the student’s perception of self and identity. Keywords: graduate employability, student award, evaluation Introduction Definitions of graduate employability have developed over time, changing in line with an increasing emphasis on individualisation, moving from Hillage & Pollard’s (1998) description of movement through the labour market through Yorke’s (2006) set of achievements improving the probability of employment; Bridgestock’s (2009) emphasis on adding career management skills and more recently Pegg, Waldock, Hendy-Isaac and Lawton’s (2012) return to a set of attributes which improve the probability of success in achieving appropriate employment with associated benefits to all stakeholders. However, despite this recent assumption of benefit to stakeholders there appears to be widespread concern that employers are not benefitting from graduate employability: that employer expectations and needs are not being met by graduates (Hinchcliffe & Jolly, 2010; Jackson, 2014; Raybould & Sheedy, 2005; Wilton, 2012). mailto:T.Wright@newman.ac.uk mailto:S.Jeffries-Watts@bham.ac.uk Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 153 Boden & Nedeva (2010) suggest that ‘employability’ in UK Higher Education (HE) is a re- badging of the traditional contributions of the HE sector to the skills and knowledge of the population, marking the transfer of authority over what constitutes employability from HE to the state, with the associated economic benefits for the state and business of requiring individuals to self-invest in education in order to ensure their individual employability. This state takeover of the employability discourse has led to an increased demand for Higher Education Institutions (HEIs) to take action to promote the employability of their graduates (Department for Business Innovation & Skills (BIS), 2009), allied to pressure on HEIs to identify how they aim to develop the employability of their graduates, and to report on associated activities and their outcomes (BIS, 2009). There is evidence that the role of grades and status of the institutions from which qualifications are attained may come into play during recruitment (Moreau & Leathwood, 2006; Tomlinson, 2008) possibly as a means of providing distinction between applicants. Indeed Tomlinson (2012) suggests that mass higher education, through such distinctions, may now be perpetuating the societal inequalities that it was designed to eliminate. It may therefore, not be unreasonable to speculate whether the role of subject grades and institutional status hold the potential to be replaced to some extent, or at least supplemented, with the arrival of additional indications of graduate employability levels. In addition Brooks & Everett (2009) note that alongside a widespread pessimism about any relationship between a degree and professional employment, there seemed to be an acceptance among recent graduates that credential inflation was inevitable and that as an individual there was a need to compensate for this by specialising and ‘gaining an edge’ over one’s competitors. These tendencies carry with them a consumer approach to HE, and imply an expectation of delivery from UK HEIs in terms of graduate employment (Brooks & Everett, 2009), rather more like expectations which may be placed upon institutions outside the UK and Europe. For example, there is a service at The University of Hong Kong called: ‘Careers and Placement’ which actively assists employers in selecting and recruiting suitable candidates (Center of Development and Resources for students). One way in which HEIs have responded to these imperatives is to set up student award schemes, the aim of which is to enhance the employability and career development of their graduates. University employability awards are proliferating both in the UK and abroad (University of Queensland Advantage Award; Swansea Employability Award; Nottingham Advantage Award), but a search of the literature appears to show little evidence of their effect, although Speight, Lackovic and Cooker (2012) note the differing understandings of the role of a university award and indeed of student employability between different stakeholders. Cleary et al. (2007), writing for the Business, Industry and Higher Education Collaboration Council in Australia, noted the lack of objective evidence available in general on the effectiveness of university interventions to equip students to cope with the modern workplace. This may be because of the difficulties inherent in attempting to measure or evaluate human experience. Nevertheless, this study is an attempt to evaluate one UK employability award. The award concerned in this study had its first completing cohort in the academic year 2003/4, and represents one large UK civic university’s means of enhancing the employability of graduates enrolled on the award. The Personal Skills Award (PSA) is the University of Birmingham’s employability award, co-curricular award, or graduate award (Nelson, Jeffries & Mann, 2013); an optional course which works outside the individual’s existing degree to develop their career related skills and attributes. It is a structured program designed for students to develop their skills, to understand and recognise skills they’ve gained from their experiences, and to teach students how to articulate their skills and experiences for the graduate recruitment process and/or application for further study. At the heart of the majority of awards is self-reflection which is crucial in terms of a student’s employability: self-reflection Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 154 acts as a vehicle for developing and realising a student’s capabilities from their engagement in extra-curricular activities (Thompson, Clark, Walker, & Whyatt, 2013) and within all areas of their life, in preparation for a complex and ever-changing world (Jackson, 2011). The Personal Skills Award aims to build a bridge between self-reflection and encouraging students to explore and engage in the opportunities around them, which in turn enhances their student experience at university. The PSA does this by formally recognising the learning students derive from hundreds of on- campus extra-curricular activities, from peer mentoring to developing a mobile app; from student society and committee involvement to engaging with a careers mentor. These opportunities encourage the development of new skills, new networks and personal interactions, as well as providing learning experiences for students. Sitting alongside the extra- curricular activities are taught skills modules, short skill sessions, and online courses, providing opportunities for students to explore skills and abilities in more depth. The compulsory workshop element of the PSA assimilates students’ engagement in the classroom and in activities and teaches them the principles of self-reflection; additionally it teaches the ‘language’ of skills so that they can articulate their experiences in a meaningful way to graduate recruiters and for entry into further study. This is the current form of the PSA and is referred to in the results section as the ‘Activities Pathway’. Although it is not now available, at the time of the commencement of this study a modular award was also offered where students took modules which taught specific skills as well as reflective practice, referred to in the results section as the ‘Modular pathway’. This skills and attributes approach to employability (CBI/ UUK, 2009; Diamond, Walkley, & Scott-Davis, 2011) assumes behaviour (performance) is objectively observable and thus may be tested via quantitative methods and this is the starting point for this study. It is important to note, however, issues raised in the literature which may mitigate against the skills and performance approach. It has been suggested that the agenda of skills and employability is not necessarily always fully shared by students; that it takes little account of other ways in which the pedagogical, organisational, cultural and social experience of HE can change the lives and values of individuals, contributing in complex ways to their aspirations post-graduation (Jary & Shah, 2009). Alternative views of graduate employability point out that it incorporates an element of interpretation of situation and the potential for practice as well as the understanding of oneself as an individual who engages in practices in a particular context, thus encompassing issues of identity as well as skills and performance. It has been proposed that during their time at university the student develops a situated identity; that of ‘graduate’ which is confirmed by their ability to be recognised as such by significant others, for example, by gaining employment at a graduate level. (Holmes, 2001). Using this model of graduate employability, it has been suggested that graduates need to think about their values, engagement and intellect as well as performance, thus incorporating potential and the idea of learning as a process of ‘becoming’ (Hinchcliffe & Jolly, 2010) in order to become more employable. Similarly, Auburn (2007) noted shifts in student identity related to placement learning and emphasised the need for a developmental understanding of the experiences of the student who becomes the graduate. More recently Jackson (2016) presents a case for ‘pre-professional identity’ in graduates, resulting from membership in, and engagement with, multiple communities of practice during their time in HE. Thus, while this study involves evaluation and measurement of skills and attributes in order to achieve the research aim, it is important to acknowledge the growing literature which suggests that this may be too simplistic and mechanistic an approach to student employability. The aim of this independent evaluation of the PSA student award at Birmingham University was to explore the extent to which the award fulfils its advertised role as an employability Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 155 enhancement intervention for students, adding value in the graduate employability market to the students who undertake it. Methodology Kirkpatrick’s (1998) model of training evaluation was used as a structure for evaluating the success of the award. The model comprises four levels and data was collected at all four levels. These were mapped onto specific project objectives as shown below: Level One Participants’ reactions. This level of evaluation focuses on participants’ satisfaction with a training program and informs the next stage of the evaluation process Project Objective: Exploring the effects, in terms of employability skills, of participating in the award as reported retrospectively by students. Level Two Assessing the impact of the course on participants’ learning. This level provides data relating to level one and two of the model Project Objectives: To explore to what extent, if any, the award increases students’ confidence in terms of aspects of their employability To explore to what extent, if any, the award increases the ability of students to recognise, reflect on and articulate their skills (to employers) Level Three Participants’ Behaviour: This level looks at how behaviour has changed in response to training. Project Objective: To explore to what extent, if any, learning on the award translates to individual behaviour change in terms of job seeking behaviour Level Four Analyse results. This level looks at the outcomes, benefits and results linked to the training. Project Objective: To explore DLHE (Destination of Leavers in Higher Education) outcome data for the cohort of award completers involved in the evaluation project. Each level’s objectives were informed by the findings of the previous level, as well as by the evaluation model. Mixed methods For each of the first three levels and their corresponding objectives, a separate mixed methodology study was undertaken, collecting both quantitative and qualitative data. The fourth level required interrogation of existing data. In all three studies quantitative data was analysed appropriately for the level and type of data available. Qualitative data was analysed using inductive thematic analysis (Braun & Clark, 2006) The aim of the qualitative data analysis was to explore and describe the impact of the award and therefore a semantic approach was taken to identify themes, focussing on the descriptions and explicit meanings communicated by participants. Issues were identified as themes (reported in bold) based on their perceived importance in addressing the research Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 156 topic. In all studies, a second researcher confirmed the extracted themes through a second analysis. Ethics All participants throughout the evaluation were informed of the independent nature of the research and their ethical rights as outlined in the British Psychological Society’s ethical guidelines (BPS, 2006). Written consent was obtained before undertaking questionnaires, and recorded verbal consent was sought immediately prior to interviews and focus groups. All participants were provided with participant information sheets and contact details of the lead researcher. For clarity each study is reported separately below. Study 1 Design In order to collect quantitative data, a survey was constructed based on the literature and in consultation with the award management team. Questions related to program aims as well as staff expectation of the award’s impact. These concerned confidence; ability to define skills; ability to identify skills gaps; ability to fill skills gaps; ability to sell skills to employers; and predicted employer perceptions of the award. As the staff team wished to establish student perceptions of the helpfulness or otherwise of interventions to improving their employability related skills, responses were recorded using the self-report survey with a Likert-type scale of 1 to 5 where 1 indicated a perceived decrease in skill or negative perceptions of helpfulness, and 5 indicated a large perceived improvement in skill or positive perception of helpfulness. As numbers in each category were too small to allow for statistical analysis, the Likert-type scale categories were then conflated to produce a binary response: reported improvement/no reported improvement. A semi-structured interview design was used to gather qualitative data relating to reactions to the award held by those who had recently completed it. The approach allowed participants to identify important aspects of their employability as it related to the award experience from a phenomenological point of view; in practice this tended to encompass ideas about participant’s employability as a whole rather than just relating to the award. The survey questions were used as the basis for the interviews, rephrased to be open in nature by replacing To what extent… in each question with How… with supplementary open interventions used to encourage exploration of each response. Participants A total of 40 individuals who had completed the award the previous academic year were involved in the first study and were recruited via opportunity sampling. This represented 38 per cent of the completers for the award in the academic year 2008/2009. Participants numbered 40 in the survey sample and ten in the interview sample, comprising self-selecting survey participants who volunteered to be interviewed. Findings Quantitative data Survey responses indicated that the greatest change was perceived by students in relation to their ability to define their skills and the least change was perceived in relation to their ability to fill skills gaps. The table below shows these results after analysis using a non-parametric Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 157 test to test the null hypothesis, that is, that respondents would perceive no significant change in their self-perceived abilities as a result of doing the award. Table 1: Percentage Responses for Questions Relating to Perceived Changes as a Result of Undertaking Award Question No Improvement Improvement P Change in ability to define skills 12% 89% p<0.001 Change in ability to identify gaps in your skills 15% 85% p<0.001 Change in ability to ‘sell’ your skills to employers 27% 73% p<0.05 Change in confidence in your employability 35% 65% NS Change in ability to fill gaps in your skills 42% 57% NS No effect Helpful Perceived impact of the employability award being on your transcript has on employers’ perceptions of you in terms of your employability 35% 65% NS Qualitative data From an analysis of the interviews the following linked main themes were identified: ‘Development of Skills’ and ‘Awareness’. Combined, these two themes depict the essential elements of employability and the award as identified by the student sample. Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 158 Figure 1: Thematic Diagram Depicting Study 1 Qualitative Findings Thus employability and the award was perceived in terms of two linked aspects:  Skills -both subject specific and transferable linked by  Awareness of these skills Application to a work setting These main themes incorporate sub themes as shown in Figure 1. Discussion of findings: Study 1 Increased awareness can be seen to impact positively on a student’s efficacy beliefs which may then result in higher employment aspirations. Work experience and Subject knowledge were both seen to feed in to Skills development, and the award fed into Awareness and Application to Setting. Space precludes detailed discussion of the qualitative findings here, but this may be found in Wright and Williamson (2010). Through a consideration of both sets of data it was possible to identify the role of the award within the development of student employability as defined by Knight and Yorke’s (2002) USEM employability model: the model used by the university careers service at the time (see Figure 2). The award can be seen as an addition to the value of a main subject degree. The main degree provides what Knight and Yorke refer to as ‘Understanding of the Discipline’ in Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 159 addition to more specialised ‘skills’. Participants see the award as a means of developing more general attributes rather than acquiring specific skills. Overall the results of Study 1 suggested that the award’s primary strength in its form at the time, lay in the opportunities and support it offered students during the development of awareness around their existing skills and the application of these skills to the employment setting. This aspect clearly maps onto the ‘meta-cognition’ aspect of the USEM module. Similarly, this function can also be seen to feed into participants’ self-efficacy, the final component of Knight & Yorke’s (2002) model. Figure 2 shows a visual representation of the award’s role in conjunction with existing experience and the main subject degree of students. Figure 2: Illustration of the Role of the Award in the Development of Employability as Defined by the USEM Model (Knight & Yorke, 2002) Results appeared to show that the award primarily impacted the sample’s ability to define their skills, as opposed to filling any identified skills gap. This finding suggested that comparisons of pre and post award employability would most usefully focus on the meta-cognition and self- efficacy aspects of employability, removing the dimension of ‘skills’ in the evaluation model. This led to the two objectives noted above at the Level Two stage of the evaluation. Study 2 Design For Study 2 a mixed method, utilising both qualitative (focus groups) and quantitative (psychometric tests) methods was used to collect data about the development of metacognition and self-efficacy in award students. Quantitative data collection involved two independent groups, award and non-award students and two administrations with a six month interval between these. This design was intended to identify any significant differences over time on two dependent variables, metacognition and self-efficacy. For numbers in each sample please see Table 2. Qualitative data collection utilised a focus group method to explore perceived learning and employability development in award students over a six month period. Thematic analysis (Braun & Clarke, 2006) was used to identify themes within the data and any change/ development after the six month period. Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 160 Participants Quantitative data collection An opportunistic sampling method was used to recruit student participants (see Table 2 for breakdown of numbers). The control (non-award) sample was drawn from attendees at the award open day who did not enrol on the award, as it was hypothesised that this sample would hold similar characteristics (an awareness of employability issues and some motivation to participate in extracurricular activities) to those who had enrolled in the award. The award sample was recruited through award workshops. Follow up questionnaires were distributed six months later via email and at award events/workshops. Attrition levels were high, with all sample groups reducing in number by just over 50 per cent due to difficulties in re-engaging students at the six month stage. Table 2: Breakdown of Participation Levels for all Groups. Baseline 6 month follow up Control group 112 58 Modular Pathway 115 50 Activities Pathway 143 64 The average age of participants for each group was between 18-21 years. The majority of participants (50%) were in their second year of study and this division between year groups was similar for all groups. Between 67 per cent and 77 per cent of each group was, or had previously been, in paid employment with the majority completing between one and16 hours per week. Qualitative data collection Focus group samples were made up of six to ten volunteers attending award workshops. A total of five groups were sampled at enrolment on the award and again after six months. The number and membership of groups were determined by the number of students who volunteered to take part in the research. A total of 71 students took part in the focus groups. 67 per cent of focus group members were female and 33 per cent were male.  Materials Psychometric tests were administered to assess students’ levels of self-efficacy and metacognition at two time periods. All measures have previously been shown to possess acceptable levels of validity and reliability (Hall, 1994; Schwarzer & Jerusalem, 1995).  Metacognition Students’ metacognition was assessed via the ‘Executive Process Questionnaire’ (Hall, 1994). The questionnaire consists of 40 statements about the participants’ approach to learning. There were four response options: ‘Almost Never’, ‘Seldom’, ‘Frequently’ and ‘Almost Always’. Scores range from one to four for each item with a possible overall score of 160. Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 161  Self-efficacy Self-efficacy was measured using the ‘Generalised Self-Efficacy Scale’ (Schwarzer & Jerusalem, 1995). This scale is a ten-item scale consisting of statements regarding the individual’s belief in their ability to respond to a given situation. Responses range from ‘Not at all True’ to ‘Exactly True’. Scores range from one to four for each item with an overall possible score of 40. The additional instruction When responding to the below questions please keep in mind issues relating to your employability (e.g. job seeking, interviews) was added in order to assess self-efficacy relating to employability. In addition to the above measures, demographic data relating to year and subject of study, age, gender, previous work experience was collected. This data allowed exploration of possible confounding variables.  Focus group questions A question set was developed for the focus group data. This question set consisted of enquiries about current perceived employability; what it means to be employable; what factors would enhance or subtract from employability; and what students hoped to or had got out of the award (in assisting them in this employability development). Open questions were used throughout which were based upon both the literature and the phase one data of this research project. Questions were reviewed by a second researcher to check relevance. Procedure Quantitative data collection Students were approached by researchers at award workshops and events. Before completing the questionnaires participants were provided with an information sheet outlining the nature and purpose of the questions and a consent form, which they were asked to sign once they indicated their full understanding of the research. Participants were asked to provide their student number to enable tracking at the second administration point. Inferential statistics were performed to investigate any potential relationships or differences between groups. Qualitative data collection Participants’ were self-selecting volunteers from the quantitative award data set who were informed of the nature and purpose of the research as well as the anonymity and confidential nature of any contributions and their right to withdraw from the process at any point. Participants were asked questions about how they perceived their current employability and if/how they expected/found the award to impact this. A number of probe questions were used to clarify and confirm responses. Each focus group was recorded with the consent of all participants included in the recording and groups lasted between 15 and 25 minutes. Recordings were transcribed and thematic analysis (Braun & Clarke, 2006), was conducted on the data by two independent researchers in order to identify themes. Results Quantitative results Before looking at comparisons of scores between the three groups, possible confounding variables were explored. Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 162  Work experience Table 3: Comparison of Mean Baseline Scores for those with Differing Levels of Hours of Work Metacognition Self-efficacy Unemployed 104.61 (9.24) 30.57 (4.55) 1-16 hours 108.09 (8.70) 31.17 (3.73) 16-24 hours 108.91 (11.08) 31.64 (3.83) > 24 hours 107.81 (8.50) 31.15 (4.33) Note: standard deviation (SD) in brackets, significant results in bold An independent samples t test was carried out comparing metacognition scores for those who were unemployed (n= 84) and those who worked up to 16 hours per week (n = 153). Caution must be exercised when interpreting these results as sample sizes are unequal. Results identified a significant difference (t (df235,162)=2.88, p= 0.004) between the groups. Findings suggest those not employed score lower in metacognition than those working up to 16 hours per week (x difference = 3.48, CI 1.10 to 5.86).  Age An investigation of the metacognition and self-efficacy scores for the three age groups (18-21, 22-29, 30 plus) showed no clear differences between participants of various ages. Table 4: Age Comparisons for Mean Baseline Scores Metacognition Self-efficacy 18-21 107.23 (9.02) 31.012 (4.13) 22-29 107.45 (12.19) 31.44 (3.23) 30 + 108.00 (8.00) 30.50 (4.04) Note: standard deviation (SD) in brackets  Baseline skills comparisons An exploration of each group’s mean baseline metacognition and self-efficacy scores indicated little difference between groups at first administration (see Table 3) and a MANCOVA controlling for the impact of hours worked confirmed no significant difference between the groups (p=0.165). Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 163 Pre and post comparisons  Work experience and skills comparison The significant difference in metacognition scores between those who worked 16-24hrs and those who did not work, evident in the baseline data, was not evident in the metacognition scores for award participants at the six month administration stage (t (df49,45) =0.76, p=0.47)  Pre and post skills comparisons An exploration of each group’s metacognition and self-efficacy scores indicated little difference between groups or scores at the six month administration point (see Table 5). Table 5: Mean scores for PSA and Control Groups at Baseline and after 6 Months Metacognition Self efficacy Baseline At 6 months Baseline At 6 months Control Group 107.32 (9.80) 110.92 (7.95) 30.59 (3.96) 29.88 (3.68) Modular Pathway 107.85 (9.14) 107.67 (10.17) 30.72 (4.37) 31.00 (5.67) Activity Pathway 106.80 (8.95) 107.53 (9.29) 31.64 (3.86) 32.39 (5.61) Note: Standard deviation (SD) in brackets A 2-way mixed MANOVA confirmed that there were no significant differences between groups or time on either scale (p=0.204) Qualitative results Figure 3: Thematic Diagram Depicting Study 2 Qualitative Findings on Award Enrolment Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 164 The three main themes above: ‘Self-development’, ‘Award as asset’, ‘Work experience’ were identified as having an impact on students participating in the award. These three themes were all identified as relating to a further theme: ‘Employability’. Figure 4: Thematic Diagram depicting Study 2 Qualitative Findings Six Months after Commencing Award The three main themes above (‘Awareness and Communication’ linked by the theme of ‘Application to employment setting’, with ‘the award as proof’ of this) were identified as having an impact on award students. The cognitive processes of the ‘strategic application of skills development’, and ‘reflection via self-assessment’, both leading to improved ‘confidence’ were also identified in the data. Again space precludes a detailed discussion of the qualitative themes within the data. This can be found in Wright (2012) Discussion of findings: Study 2 The non-significant results over time in the quantitative data from Study 2 taken alongside some clear differences over time in the qualitative data suggest that the award, rather than increasing student’s cognitive skills, may be changing their application, and the meaning that is ascribed to their possession by those who undertake the award. The lack of any significant differences over time or group tend to suggest that the award itself has no effect on participant’s metacognition or self-esteem in a quantitative sense. However, the disappearance of the difference in metacognition scores for award participants at the six Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 165 month point between those who worked part time and those who did not, suggests that perhaps the award has some effect on metacognitive ability which levels out the advantage conferred by part time work. Both sets of qualitative data show that focus group participants were able to demonstrate reflective thinking, but those in the later groups showed a greater degree of discrimination and an awareness of a developmental process which was not present in the themes at enrolment on the award. The comments describing the process of reflection, application of experience to context and a resulting increase in confidence may link to the concept of learning as part of a process of ‘becoming’ (Hinchcliffe & Jolly, 2010). This complex cognitive process may be difficult to measure, thus we find non-significant quantitative results alongside clear differences in qualitative data. The description of a process, in terms of how award students describe themselves and their ability to learn and grow, may also link to the graduate identity approach to employability as advocated by Holmes (2001) or perhaps the award may be considered as a contributing community of practice in Jackson’s (2016) pre-professional identity model. Study 3 Design Study 3 involved a survey design, collecting numeric and individual data about job seeking behaviour. A survey was devised, based on the framework suggested in Kirkpatrick (1998), to explore changes in job seeking behaviour after undertaking the award. Participants An opportunistic sampling method was used to recruit participants from those who had completed the award in the year 2011. A total of 64 students completed surveys, which was 34 per cent of the total number who had completed the award that year. Thus the same cohort as was involved in the Stage 2 data collection was surveyed at completion of their award. Procedure In the first instance all award completers in the academic year 2010/2011 were emailed an electronic copy of the survey with a request to complete and return to the research team by email. To supplement this, participants were recruited at the social event which accompanied the awards ceremony and consenting individuals were asked to complete a paper copy of the survey. The surveys were than analysed via descriptive statistics which were illustrated by the individual data supplied freehand in answer to specific questions about behaviour change. Findings Eighty-four per cent of respondents said that they were now implementing employability related behaviours that they were not before undertaking the award. When asked to elaborate on these, respondents described reflecting on their skills and experiences; applying skills and attributes to specific situations and contexts; increased ability to articulate skills and attributes, and increased confidence. This is evidenced in the following comments: I have learnt to reflect and expand on previous work/life experiences, which has resulted in enabling me to express/convey myself better. Thinking more about how to effectively use my time outside of academic studies to put myself at an advantage over other graduates. Also I now feel more confident that I can analyse what I’ve done and how it is relevant to particular positions. Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 166 I have learnt more about myself and I’m now trying to "play to my strengths" rather than focus on my weaknesses. This is affecting my career decisions. Fifty one per cent of respondents also planned to change their behaviours in the future as reflected in comments such as: Tailoring my CV/cover letters to specific jobs. To develop my commercial awareness. Be more proactive. Further freehand comments were invited in a section which asked respondents if there was anything else they would like to comment on about how the award had altered their behaviour. There were two strands of responses here: a. Those who reported specific improvements in attributes directly related to the seeking of employment, who commented, for example: It's been useful for improving my confidence and networking skills. Improved my CV and the ability to recognise and develop my transferable skills. b. Those who reported somewhat more global effects on the self, such as self-improvement, development; preparedness, growth, with comments such as: The award has prepared me for life beyond my degree. It has helped me grow as a whole in the sense of employability. I learned a lot. I changed my views and thinking. Discussion of findings: Study 3 The Stage 3 results show evidence of behaviour change as a result of learning on the award, but also seem to demonstrate changes in self- concept in some respondents. The idea of preparedness for the future was mentioned specifically by two respondents and when taken in conjunction with the more numerous comments about feelings of confidence about completers’ abilities to manage future challenges, implies a sense of improved resilience or resourcefulness in respondents. The concept of career adaptability may be helpful here: Savickas (1997) notes the complexity of vocational behaviour across a multiplicity of contexts and with diverse groups of people, and defines career adaptability as: the readiness to cope with the predictable tasks of preparing for and participating in the work role and with the unpredictable adjustments prompted by changes in work and working conditions (p.254). Adaptability resources are the self-regulation capacities that an individual may draw upon to solve the unfamiliar, complex and ill-defined problems presented by developmental vocational tasks, occupational transitions and work traumas (Savickas, 1997). It is therefore possible to conceive of the award as developing the career ‘adapt-abilities’ (Savickas & Porfeli, 2012) of students, thus enhancing both their readiness and their resources to deal with the changing world of work and to begin to construct their own individual careers. Further work in this area would be required, however, to draw any firm conclusions. Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 167 Study 4 Procedure In the UK there is a legal requirement for all HEIs to collect and report to the UK Higher Education Statistics Agency on the number of graduates in graduate level employment six months after graduation. This national survey, the Destination of Leavers in Higher Education (DLHE), is published and forms one of the measures used in national league tables which purport to compare the relative merits of different UK universities. It is thus an important performance indicator for UK university careers providers as well as for the universities. The destinations data for award completers was obtained from the university in the study and was compared to the destinations data for the university overall in the same academic year. Findings The award cohort (n=188) showed an 86 per cent graduate level of employment as opposed to 82 per cent for the university as a whole (including the award students) in the destinations data for the year in question. Discussion of findings: Study 4 The data was limited in nature and took some time to obtain, due to restrictions on its usage. Thus its usefulness to this study is also somewhat limited as it was not possible to isolate different groups of students from the data set owned by the university in question. The award cohort was smaller by several hundred degrees of magnitude than the comparator figure which was available, therefore direct comparison was not possible. Neither is it possible to know whether those who undertook the student award were those who might have achieved graduate level employment within the first six months due to factors other than undertaking the award. However, the limited data available does tentatively suggest that completion of the student award may contribute to a slightly greater probability of gaining graduate employment in the first six months. Discussion: Project as a whole There were a number of limitations with the studies. In Stage 2, due to the data collection methods, a degree of standardisation necessary for optimal use of psychometric testing was absent. This may have impacted adversely on the accuracy of the quantitative data. However, data was collected in the same way at both time points. A further limitation on the quantitative data was the significant attrition rate between the two administrations of the tests, as sample sizes reduced by 50 per cent between tests. Similarly, the participants in the Stage 2 focus groups were self-selecting rather than being randomly selected. This may impact on the data gathered in that those who volunteer in any setting tend to be those individuals who are most enthusiastic and motivated, thus skewing the data. However, as both sets of focus group participants were self-selected, both sets of data may be equally skewed and it is differences between the groups over time which are reported. At Stage 3 a proportion of participants were recruited from the award ceremony, again holding the possibility of a skewed sample of enthusiastic and motivated individuals, producing a picture which does not reflect the award completion group as a whole. However, as the award is a voluntary exercise, all those individuals who complete it could, by definition, be described as motivated and enthusiastic. Outcome data was supplied by the university and was very limited due to constraints upon usage of Destinations of Leavers in Higher Education data. Outcome data was for those who completed the award only. No information was available for a substantial minority who undertook part of the award but did not complete, who presumably would experience some effect, but who were excluded from the data set. However, outcome data for subsequent years Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 168 continue to show small differences in employment rates with award completers in the 2014/15 academic year (the most recent year for which figures are available) showing an 87 per cent graduate level employment rate as opposed to 85 per cent for the university as a whole. Tymon (2013) suggests that communication skills and self-confidence may be developed as part of the process of undertaking a degree, and this suggestion may go some way towards explaining the lack of difference found in either metacognition or self-esteem scores in Study 2. However, the increased aspiration reported in Study 1 would seem to imply some change in perceived confidence or competence in award participants which was not measured by the psychometric measures used, but which appeared clearly in the qualitative findings in Studies 2 and 3 The disappearance at the six month stage of award study of the difference found in metacognition scores between those who had experience of the workplace and those who did not suggests that there may be a similar advantage conferred by award study to that conferred by experience of the workplace; Adams and Hancock (2000) noted that relevant work experience predicted greater success at Master of Business Administration study than did any other factor including grade point average. The reported changes in employability related feelings and behaviours throughout the linked studies imply a sense of improved resilience or resourcefulness in respondents. It is therefore hypothesised that the award may enhance student career adaptability (Savickas, 1997), via development of their psychosocial resources to negotiate the transitions and challenges of life and work, producing an internal change in the student’s perception of self. It is also possible that the award scheme may encourage a shift in graduate identity which is then confirmed and recognised by the granting of an award by the university, thus enabling the student to more confidently approach an ‘agreed identity’ as explicated in Holmes, 2013, or, as proposed earlier, it may contribute to a graduate’s pre-professional identity (Jackson, 2016). As Tomlinson (2012) notes, how employable a graduate is tends to be subjective and can be dependent on their perception of themselves in relation to the world of work. Conclusion Overall findings indicate that employability related values, attitudes and behaviour may all change in students as a result of the award’s experience, which tends to suggest a more complex and holistic process takes place over award study than simple skills acquisition. This study suggests that award experience holds the potential to enhance early graduate self- perception in relation to their work related dispositions. Further studies across institutions would be helpful to ascertain whether these findings are general, or a function of this particular program. Quantitative study of particular cohorts of students in the destination data of universities would help in targeting groups of students who may benefit from award study, and longitudinal study following the lives and careers of award completers would give a clearer indication of any long term benefit. Acknowledgements The authors’ thanks and gratitude go to Joan Cartledge and the PSA team. Thanks are also due to Stella Williams for her valuable work in the early stages of the project. Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 169 References Adams, A.J., & Hancock, T. (2000). Work experience as a predictor of MBA performance. College Student Experience, 34(2), 211–217. Auburn, T. (2007) Identity and placement learning: Student accounts of the transition back to university following a placement year. Studies in Higher Education, 32(1), 117–133. doi: 10.1080/03075070601099515 Boden, R., & Nedeva, M. (2010). Employing discourse: Universities and graduate ‘employability’. Journal of Education Policy, 25(1), 37–54. doi: 10.1080/02680930903349489 Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3, 77–101. doi: 10.1191/1478088706qp063oa Bridgstock, R. (2009). The graduate attributes we’ve overlooked: Enhancing graduate employability through career management skills. Higher Education Research and Development, 28(1), 31–44. doi: 10.1080/07294360802444347 British Psychological Society. (2006). Code of Ethics and Conduct. Leicester: British Psychological Society. Brooks, R., & Everett, G. (2009). Post-graduation reflections on the value of a degree. British Education Research Journal, 35(3), 333–349. doi: 10.1080/01411920802044370 CBI and Universities UK (2009) Future fit: Preparing graduates for the world of work. Retrieved from: http://www.universitiesuk.ac.uk/Publications/Documents/FutureFit.PDF Center of Development and Resources for Students, The University of Hong Kong. http://cedars.hku.hk/showext.php?id=cp Cleary, M., Flynn, R., Thomasson, S., Alexander, R., & McDonald, B. (2007). Graduate employability skills: Prepared for the Business, Industry and Higher Education Collaboration Council. Canberra: Dept of Education, Science and Training. Department for Business Innovation & Skills (BIS). (2009). Higher ambitions: The future of universities in a knowledge economy. Retrieved from: http://www.employability.ed.ac.uk/documents/Staff/PoliciesReports/BIS- HigherAmbitions-Summary-Nov2009.pdf Diamond, A., Walkley, L., & Scott-Davies, S. (2011). Global graduates into global leaders. Report: Association of Graduate Recruiters, CFE Research & Council for Industry & Higher Education. Retrieved from: http://www.ncub.co.uk/reports/global-graduates-into- global-leaders.html Hall, C. (2001). A measure of executive processing ability in college students. College Student Journal, 35(3), 442–451. Hillage, J., & Pollard, E. (1998). Employability: Developing a framework for policy analysis. London: Department for Education and Employment. Hinchcliffe, G., & Jolly, A. (2010). Graduate identity and employability. British Education Research Journal, 37(4), 563–584. doi: 10.1080/01411926.2010.482200 Holmes, L. (2001). Reconsidering graduate employability: The Graduate identity approach. Quality in Higher Education, 7 (2), 111–119. doi:10.1080/13538320120060006 Holmes, L. (2013). Realist and relational perspectives on graduate identity and employability: A response to Hinchliffe and Jolly. British Educational Research Journal, 39(6), 1044–1059. doi: 10.1002/berj.3022 Jackson, D. (2014). Testing a model of undergraduate competence in employability skills and its implications for stakeholders. Journal of Education & Work, 27(2), 220–242. http://www.universitiesuk.ac.uk/Publications/Documents/FutureFit.PDF http://cedars.hku.hk/showext.php?id=cp http://www.employability.ed.ac.uk/documents/Staff/PoliciesReports/BIS-HigherAmbitions-Summary-Nov2009.pdf http://www.employability.ed.ac.uk/documents/Staff/PoliciesReports/BIS-HigherAmbitions-Summary-Nov2009.pdf http://www.ncub.co.uk/reports/global-graduates-into-global-leaders.html http://www.ncub.co.uk/reports/global-graduates-into-global-leaders.html Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 170 Jackson, D. (2016). Re-conceptualising graduate employability: The importance of pre- professional identity. Higher Education Research and Development, 35(5) 925–939. doi: 10.1080/07294360.2016.1139551 Jackson, N. (2011). The lifelong and lifewide dimensions of living, learning and developing. In N.J. Jackson (Ed.), Learning for a complex world. A lifewide concept of learning, education and personal development. Authorhouse: Bloomington, USA. Jary, D., & Shah, T. (2009). What is learned at university? SOMUL Working Paper 6. Retrieved from: https://www.open.ac.uk/cheri/documents/WorkingPaper6.pdf Kirkpatrick, D. L. (1998). Evaluating training programs (2nd Edition). San Francisco: Berrett- Koehler. Knight, P., & Yorke, M. (2002). Employability through the curriculum. Tertiary Education and Management, 8, 261–276. Retrieved from: http://www.qualityresearchinternational.com/esecttools/esectpubs/Employabilitythrough thecurriculum.pdf Moreau, M-P., & Leathwood, C. (2006). Graduates employment and the discourse of employability: A critical analysis. Journal of Education & Work, 19(4), 305–324. doi: 10.1080/13639080600867083 Nelson, D., Jeffries, S., & Mann, V. (2013). Extra-curricular awards stimulus papers: Association of Graduate Careers Advisory Services (AGCAS) perspective. Retrieved from Quality Assurance Agency: http://www.qaa.ac.uk/en/Publications/Documents/extra-curricular-awards-AGCAS.pdf Nottingham Advantage Award. http://www.nottingham.ac.uk/careers/students/advantageaward/index.aspx Pegg, A., Waldock, J., Hendy-Isaac, S., & Lawton, R. (2012). Pedagogy for employability. York: Higher Education Academy. Retrieved from: https://www.heacademy.ac.uk/sites/default/files/pedagogy_for_employability_update_2 012.pdf Precision Consultancy (2007). Graduate employability skills. Commonwealth of Australia: ACT. Raybould, J., & Sheedy, V. (2005). Are graduates equipped with the right skills in the employability stakes? Industrial & Commercial Training, 37(4/5), 259–263. doi: 10.1108/00197850510609694 Savickas, M. L. (1997). Adaptability: An integrative construct for life-span, life-space theory. Career Development Quarterly, 45(3), 247–259. doi: 10.1002/j.2161- 0045.1997.tb00469.x Savickas, M.L., & Porfeli, E.J. (2012). Career Adapt-Abilities Scale: Construction, reliability, and measurement equivalence across 13 countries. Journal of Vocational Behavior, 80(3), 661–673. doi: 10.1016/j.jvb.2012.01.011 Schwarzer, R., & Jerusalem, M. (1995). Generalized self-efficacy scale. In J. Weinman, S. Wright & M. Johnston, Measures in health psychology: A user’s portfolio. Causal and control beliefs (pp. 35–37). Windsor, England: NFER-NELSON. Speight, S., Lackovic, N., & Cooker, L. (2012). Stakeholder attitudes towards employability in a Sino-British university. Journal of Teaching and Learning for Graduate Employability, 3(1), 26–40. doi: http://dx.doi.org/10.21153/jtlge2012vol3no1art556 Swansea Employability Award. http://www.swansea.ac.uk/employability-awards/ https://www.open.ac.uk/cheri/documents/WorkingPaper6.pdf http://www.qualityresearchinternational.com/esecttools/esectpubs/Employabilitythroughthecurriculum.pdf http://www.qualityresearchinternational.com/esecttools/esectpubs/Employabilitythroughthecurriculum.pdf http://www.qaa.ac.uk/en/Publications/Documents/extra-curricular-awards-AGCAS.pdf http://www.nottingham.ac.uk/careers/students/advantageaward/index.aspx https://www.heacademy.ac.uk/sites/default/files/pedagogy_for_employability_update_2012.pdf https://www.heacademy.ac.uk/sites/default/files/pedagogy_for_employability_update_2012.pdf http://dx.doi.org/10.21153/jtlge2012vol3no1art556 http://www.swansea.ac.uk/employability-awards/ Wright, T. & Jeffries-Watts, S. (2017). Does it do what it says on the tin? – Evaluation of an established UK university employability enhancement award. Journal of Teaching and Learning for Graduate Employability, 8(1), 152–171. 171 Tomlinson, M. (2008). 'The degree is not enough': Students perceptions of the role of higher education credentials for graduate work and employability. British Journal of Sociology of Education, 29(1), 49–61. doi: 10.1080/01425690701737457 Tomlinson, M. (2012). Graduate employability: A review of conceptual and empirical themes. Higher Education Policy, 25(4), 407–431. doi:10.1057/hep.2011.26 Thompson, L. J., Clark, G., Walker, M., & Whyatt, D. J. (2013). ‘It’s just like an extra string to your bow’: Exploring higher education students’ perceptions and experiences of extracurricular activity and employability. Active Learning in Higher Education, 14(135), 135–147. doi: 10.1177/1469787413481129 Tymon, A. (2013). The student perspective on employability. Studies in Higher Education, 38(6), 841–856. doi: 10.1080/03075079.2011.604408 University of Queensland Advantage Award. http://www.uq.edu.au/advantage-award/ Wilton, N. (2012). The impact of work placements on skills development and career outcomes for business and management graduates. Studies in Higher Education, 37(5), 603–620. Wright, T. (2012). Personal Skills Award Evaluation Project – Stage 2 & 3 Report. Report for Careers Network, University of Birmingham. Wright, T., & Williamson, S. (2010). Personal Skills Award Evaluation Project – Stage 1 Report. Report for Careers Network, University of Birmingham. Yorke, M. (2006). Employability in higher education: What it is and what it is not. Learning and Employability Guides, UK: HEA. http://www.uq.edu.au/advantage-award/