Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 222 The Journal of Teaching and Learning for Graduate Employability ISSN: 1838-3815 (online) Journal Homepage: https://ojs.deakin.edu.au/index.php/jtlge/ Discipline predicts Work Integrated Learning (WIL) practice in Science courses Susan Rowland1 and Daniel Blundell2 Corresponding author: Susan Rowland (s.rowland1@uq.edu.au) 1 Professor, Faculty of Science and School of Chemistry and Molecular Biosciences, The University of Queensland, St Lucia, QLD, Australia. 2Senior Data Analyst, Faculty of Science, The University of Queensland, St Lucia, QLD, Australia Abstract Australian mathematics and science students have low participation in WIL, posing implications for student employability. To better understand this problem, an examination of the industry-placement and coursework-incorporated WIL offered across the Faculty of Science at a large research-intensive university was undertaken. The aim of the study was to provide an evidenced discussion of the types and amounts of WIL that different disciplines offer their students. A matrix was used to measure the inclusion of WIL activities in 265 courses (units of study) across all undergraduate programs in a Faculty of Science. The results, which show comparisons between disciplines, year levels, and class sizes, indicate that a high proportion of courses incorporate WIL, but that some disciplines are significantly more likely to incorporate WIL than others. This study provides important insights into how science students in different disciplines and in different levels of their degree are prepared for the workplace. In considering how to address graduate employability through integrating WIL in university STEM coursework, this study provides evidence-based justification to initiate reflection about pedagogy. Keywords: Science curriculum, employability, work- integrated learning, workplace learning Introduction Like universities in many countries, Australian universities are under pressure to demonstrate their value to students and society. A series of Government policy reforms around university funding and students’ financial contributions have commodified enrolments and student satisfaction (Loomes, Owens, & McCarthy, 2019; Tehan, 2019; Wellings et al., 2019). These ‘vectors of change’ (Krause, 2020) have increased the pressure on universities to improve their students’ satisfaction with their educational investment. An important contributor to student satisfaction is the employment rate of graduates (The Social Research Centre, 2018), an area of particular concern for universities that offer science programs. In Australia, science graduates take a disproportionately long time to find full-time work (Graduate Careers Australia, 2015; Norton & Cakitaki, 2016). Despite this, over 30% of science, technology, engineering, and mathematics (STEM) employers have difficulty recruiting appropriately-qualified https://ojs.deakin.edu.au/index.php/jtlge/ mailto:s.rowland1@uq.edu.au Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 223 STEM graduates (Deloitte Access Economics, 2014). A key contributor to this problem is the lack of transferable skills, workplace awareness, and workplace experience on the part of science graduates (AIG, 2016). Universities respond to this need in science (and other disciplines) by including work integrated learning (WIL) in the curriculum (Universities Australia, 2017). Patrick et al. (2008) describe WIL as approaches and strategies that integrate theory with the practice of work within a purposefully designed curriculum (p. 9). There is debate, however, as to what ‘counts’ as WIL. It is commonly accepted that WIL includes field-work, job shadowing, and internships (Orrell, 2011), either in the workplace or on campus in a way that emulates key aspects of the workplace (Beard & Wilson, 2006, p. 205). These important activities (which are termed ‘traditional’ WIL in this study) are frequently work proximal and can also provide highly-authentic experiences of the workplace (Kaider, Hains-Wesson, & Young, 2017; Oliver, 2015). They are not, however, always accessible to students. There are strong arguments for using a broader conception of WIL for science students – one that sits outside the placement/internship model. Non-placement WIL allows more students to participate, reduces the impost on industry, and can adapt to the constraints imposed by Covid19. Universities are working in this space and are increasingly enacting WIL through new models that include curricular and co-curricular micro-placements, online industry interactions, hackathons and competitions, and consulting opportunities (Kay et al., 2019). As the nature of work itself changes to become more flexible, more online, and more ‘gig’- constructed, we are also now changing our understanding of the workplace. Modern workers often operate in an online, but hyper-connected space in which they support rapid-fire, interdisciplinary, team-built projects. Indeed, the idea of a fixed ‘workplace’ is disappearing for many workers. WIL practice needs to change, and is changing, to reflect these realities (Dean & Campbell, 2020; Winchester-Seeto & Piggott, 2020). Students’ interaction with industry should also not be limited by discipline. Most science graduates find long-term work in non-science fields (Office of the Chief Scientist, 2020). Thus, familiarity with non-science workplaces, work practices, and work cultures is both desirable and appropriate for science graduates as they develop their employability (Small, Shacklock, & Marchant, 2018 and references therein; Rowland, Gannaway, Pedwell, et al., 2020). The most recent meta-analysis of Australian university WIL in science (Edwards, Perkins, Pearce, & Hong, 2015) showed that placements are not the only way that science students experience WIL. Universities deliver science WIL through integrating theory with the practice of work, engaging with industry and community partners, using planned authentic activities, and making purposeful links between work, the curriculum and assessment. This activity happens both on and off campus using simulations, projects, placements, and other forms of interaction with employers. Edwards et al. used their study to develop a rubric that characterizes these overt WIL activities and to highlight other types of WIL that may not fit within the ‘traditional’ WIL envelope. Their rubric examines all possible types of WIL, from those ‘traditional’ practices that are highly authentic and work proximal, to those that rely solely on information provision (Kaider et al., 2017; Oliver, 2015). Some of these low-authenticity, work-distant WIL practices could also be classed as employability or career-learning pedagogies. For the purposes of this study, and according to the work of Edwards et al., Kaider et al., and Oliver, all these activities are classed as WIL. A modified version of this tool, the Western WIL rubric, has recently been used to examine the distribution and types of WIL offered in science across an entire university (Jones, Millar, & Chuck, 2019). Importantly, this rubric identifies ‘hidden’ course-based WIL – WIL activities that are either not assessed and/or not explicitly identified to the university or to students (Jones et al., 2019, p. 352). Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 224 Although these activities are ‘hidden’ elements of the curriculum, they are still important. The inclusion of these activities in the curriculum helps students link their education to their futures. By including them, educators signal that the material being taught is relevant to the real world and has employability value to students. Anecdotal evidence suggests there are differences in the ways science disciplines prepare their students for the workplace, but we have little empirical evidence about the distribution of WIL, particularly at the science discipline level. Jones et al. (2019) showed some evidence of year-level and discipline-dependent differences in the WIL offered at one university. Radloff and Coates (2010) showed that the natural and physical sciences appear to be the least likely discipline group to offer placement and project WIL to students. Edwards et al. (2015) showed that only one in seven students enrolled in natural and physical sciences programs does an industry-based project, while the number that completes an industry placement or internship is ‘almost negligible’ (p. 52). Most recently, Universities Australia (2017) collected data from all Australian universities to quantitate WIL activity that demonstrates ‘a clear link to a workplace or employer’. The data showed that, when WIL is offered in the form of placements, projects, fieldwork, and simulations, over 55% of students in agriculture, environmental and related studies experience WIL. In contrast, only 27% of students studying natural and physical sciences get WIL exposure. Most of the work cited above used only two broad categories of science discipline – ‘Agriculture, environmental and related studies’ and ‘Natural and physical sciences’. While the data paint a big picture, they do not allow us to look closely at differences in science WIL offerings at the level of specific discipline, class size, or undergraduate year. A more detailed understanding of WIL-related practice and its influencing factors has ramifications that will help us consider how science students in different disciplines and at different levels of their degree are prepared for the workplace. This study will help provide this understanding and an evidence base for WIL-related discussions. There are two aims central to this study. The first is to develop a picture of the incidence and types of WIL offered in all of the Faculty of Science courses at a large research-intensive university (throughout this paper the term ‘course’ is used to refer to a single unit of study or subject). In particular, this study uses discipline lens to examine whether there is a difference in the amount, type, and intent of WIL offered to the students. This leads to the second aim, which is to inform the conversation about WIL in science more broadly by developing and understanding of what types of WIL are offered in science curricula. This knowledge will broaden the perspectives of what a community of science academics considers to be WIL. It will also help to conceptualise how WIL can be delivered in all science courses, rather than in the select few that offer placements and projects. The following questions are addressed in this paper: 1. To what extent are WIL activities occurring in Faculty of Science courses in the University of Queensland (UQ)? 2. What does this WIL look like, in terms of intent and pedagogy? 3. How do the activities and objectives of Work Integrated Learning differ within Faculty of Science courses by class size, year level, and discipline? Methodology Context The study was conducted at UQ, a large, public, Australian, research-intensive university. The Faculty of Science includes six multidisciplinary Schools (Departments): Agriculture and Food Science (SAFS); Biological Science (SBS); Chemistry and Molecular Biosciences (SCMB), Earth and Environmental Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 225 Science (SEES); Mathematics and Physics (SMP); and Veterinary Science (SVS). The School of Biomedical Science (SBMS in the Faculty of Medicine) also offers courses for Faculty of Science students. In this study, the term ‘Faculty of Science’ refers to the Faculty, its six Schools, and SBMS. The Faculty of Science directly enrolls over 5,600 undergraduate students in 16 degree programs at two campuses. The city-based campus houses SBS, SCMB, SEES, SMP, and SBMS programs. The rural campus houses the SAFS and SVS programs. This study addresses WIL in all undergraduate programs at both campuses. Ethics The UQ Human Research Ethics Committee granted approval for this study (# 2017000561). Participants and sample The sample size and participants were limited to course offerings within the UQ Faculty of Science. The selected courses satisfy all of the following criteria: 1. Courses for undergraduate students offered at UQ through a Faculty of Science School from July 2016 through July 2017. 2. Courses offered for students who are progressing in a Faculty of Science-controlled program (as opposed to ‘service’ courses for students completing a non-Science program). 3. Where a course was offered in duplicate or triplicate (i.e., in multiple semesters each year), the semester offering with the largest enrolment was chosen. A ‘course’ is a single unit of study (also known as a unit, a subject, or a paper at other universities). Multiple courses in combination comprise a Degree or Diploma program. Of the 656 courses available in the study period, 393 satisfied the selection criteria. The academic staff member coordinating each course was asked to complete the survey. Survey instrument The Western WIL Rubric (Jones et al., 2019) was used to collect data. The survey was tested with three academic staff from the UQ Institute for Teaching and Learning Innovation (ITaLI) and two academic staff from the Faculty of Science, after which minor modifications were made to the survey to clarify terms for use in the local context. The survey is based on the premise that there can be up to six Objectives for delivering WIL in a course (Table 1), and that these can be achieved by utilizing up to five categories of Activity (Table 2). Each Objective and Activity category can be enacted through one or more specific pedagogies (see Appendix 4, Table S1), of which there are 55 (note each pedagogy has an assigned code). Each pedagogy in Table S1 was addressed with one question in the survey (See Appendix 1). Respondents were asked whether or not the pedagogy happens in their course and a ‘yes/no’ response was solicited. All pedagogies were considered to be of equal value in the final survey score calculation. Each course gained a score from 0 to 55, reflecting the number of ‘yes’ answers. Higher scores indicate more WIL pedagogies occur in the course. Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 226 Table 1: The Possible Objectives of Delivering WIL in Courses (Edwards et al., 2015) OBJECTIVE 1 To develop curriculum linked STEM workplace/occupation specific skills and knowledge, and be able to adapt and apply them OBJECTIVE 2 To build an understanding of the nature of industry and the roles of different occupations as they relate to industry OBJECTIVE 3 To facilitate self-understanding OBJECTIVE 4 To train professionals to enter a specific STEM industry in accordance with the standards of a defined industry OBJECTIVE 5 To develop employability and contextualised language, literacy and numeracy skills OBJECTIVE 6 To develop career management skills Table 2: The Activity Groups of Educators who Deliver WIL (Edwards et al., 2015) Code Activity T ‘Show & Tell’ in-classroom S ‘Sell’ in-classroom E ‘Engage’ in-classroom P ‘Practice’ in-classroom OC ‘Out-of-classroom’ The instrument has limitations. Many questions are double-barreled (e.g., one asks if a particular activity involves ‘reflection and debriefing’); these questions may elicit negative responses if only one of the pedagogies is used. This may lead to under-reporting of WIL activity. The survey does not ask how frequently each pedagogy occurs in the course. The instrument does not make any attempt to quantitate the impact of the pedagogies on students, and this aspect of teaching and learning is outside the scope of this study. Data collection For courses in Semester 2, 2016, Summer Semester (2016-2017), and Semester 1 (2017) data collection was initiated by emailing instructors who coordinated in-sample courses and inviting them to complete the online survey. In the six months after the start of data collection (after the beginning of each semester) three reminder emails were sent to course coordinators. Non- participating coordinators were telephoned, with an offer to help them complete the survey. A completed or partially completed set of answers for a course was counted as a valid response. Of a total 393 courses, 265 responded (67% response rate, which is relatively high for an online survey (Nulty, 2008)). For the analysis, courses were classified by discipline into seven clusters: 1. Agricultural and Food Sciences (AFS); 2. Biological and Marine Sciences (BIM); 3. Biomedical Science (BIOM); 4. Chemistry and Molecular Biosciences (CMBS); 5. Earth and Environmental Sciences (EES); Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 227 6. Mathematics and Physical Sciences (MPS); and 7. Veterinary Science (VETS). Details of course assignment to clusters are given in Appendix 2. Data analysis Data analysis was performed using three approaches: 1) Exploratory data analysis; 2) Linear regression; and 3) Pairwise regression. Details of investigations around incomplete and missing data, response bias, and approaches to regression are given in Appendix 2. Results The discussion of the results is framed around the research questions which are: RQ1: To what extent are WIL activities occurring in Faculty of Science courses at UQ? RQ2: What does this WIL look like, in terms of intent and pedagogy? RQ3: How do the activities and objectives of Work Integrated Learning differ within Faculty of Science courses by class size, year level, and discipline? To address the research questions, descriptive statistics were calculated for the course WIL scores. These were used to explore: (i) WIL scores (total and individual activities) by category (Discipline, Year Level and Class Size); and (ii) WIL adoption rates overall and by Activities. A high proportion of courses incorporate some type of WIL Overall, the adoption rate for WIL was very high (Table 3). Of the 265 courses responding, 244 (92%) adopted one or more of the 55 WIL pedagogies. Only 21 courses (8%) had a zero WIL score. There was an adoption rate of at least 83% for each Activity, except for the Out-of-classroom Activity category, which was a clear outlier (74 courses, 28% adoption). Table 3: The Percentage of Courses Adopting WIL by Activity Group Activity Group Adoption Rate (%) Show & Tell 87 Sell 83 Engage 88 Practice 85 Out-of-classroom 28 WIL Score Total 92 In terms of Activities, the following scores were attained in order: Show & Tell (M = 4.58 pedagogies per course, SD = 3.05 pedagogies per course), Practice (M = 3.93, SD = 2.93), Engage (M = 3.76, SD = 2.58), Sell (M = 2.54, SD = 1.91) and Out-of-classroom (M = 0.90, SD = 2.41). On average, courses attained a WIL score of 15.7 (SD = 10.6, n = 265). Thus, on average, each course adopted 15 to 16 of the 55 WIL pedagogies. The five most-commonly-used pedagogies were: O1T2 – Explicit examples in lectures & course notes of skills relating to work (195 instances); Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 228 O1S1 – Explicit focus on why concepts, skills & information are important to work & how they may be applied during work (189 instances); O1E2 – Inquiry-based learning with explicit activities linking skills & knowledge to work (173 instances); O4T2 – Lecturer shares their own industry experience (157 instances); and O5T1 – Recognition & use of professional behaviours & artefacts (150 instances). These results show that, overall, most courses in the sample (92%) adopted some form of WIL, and that some pedagogies were much more commonly used than others. It was most common in courses to explain how and why the skills learned in the course related to work. Inquiry-based learning with links to work was also very common. Out-of-classroom WIL, where the students have direct access to a non-campus learning experience, was the least frequent type of WIL activity. These results address RQ1 and RQ2. The extent and type of WIL differs by discipline After establishing that some types of WIL are more common than others, the types and extent of WIL offered by each discipline were examined (see Appendix 4, Table S2). VETS has the highest Total WIL score (Figure 1a) and this discipline stands out nominally from the other disciplines for all activities. The next highest disciplines nominally were EES and AFS. The lowest scoring discipline overall, and in each activity, was MPS (Figure 1a–e). The VETS Total WIL Score was significantly higher than all disciplines except EES. The EES Total WIL score was similar to the AFS score, and was significantly higher than scores for BIM, CMBS, and MPS (see Figure 1a). This trend was seen for all WIL activity types, other than Out-of-classroom. VETS is the only discipline with a relatively high score for Out-of-classroom activity, but due to small course counts the 95% confidence interval is wide and it shows overlap with the lower scores in the other disciplines (Figure 1f). Findings of importance are:  VETS had the highest Show & Tell, Sell, Practice, and Engage WIL scores. In each case the VETS score was significantly higher than the scores for BIM, BIOM, CMBS, and MPS.  EES had significantly higher scores than the BIOM, CMBS, and MPS for the Show & Tell, Sell, Practice, and Engage categories (Figure 1b–e).  AFS scores for Sell were significantly higher than BIOM, CMBS, and MPS (Figure 1c). AFS also gained significantly higher Show & Tell scores than BIM, CMBS, and MPS (Figure 1b). These results show that there are differences between the WIL offerings for students. They address RQ2 and RQ3. Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 229 0 10 20 30 Veterinary Science Mat hematics and Physical Sciences Earth and Environmental Sciences Chemistry and Molecular Biosciences Biomedical Science Biological and Marine Sciences Agricultural and Food Sciences WIL Score WIL Total(a) 0 2 4 6 8 10 Veterinary Science Mat hematics and Physical Sciences Earth and Environmental Sciences Chemistry and Molecular Biosciences Biomedical Science Biological and Marine Sciences Agricultural and Food Sciences Score Show & Tell(b) 0 1 2 3 4 5 Veterinary Science Mat hematics and Physical Sciences Earth and Environmental Sciences Chemistry and Molecular Biosciences Biomedical Science Biological and Marine Sciences Agricultural and Food Sciences Score Sell(c) 0 2 4 6 8 Veterinary Science Mat hematics and Physical Sciences Earth and Environmental Sciences Chemistry and Molecular Biosciences Biomedical Science Biological and Marine Sciences Agricultural and Food Sciences Score Practice(e) 0 2 4 6 8 Veterinary Science Mat hematics and Physical Sciences Earth and Environmental Sciences Chemistry and Molecular Biosciences Biomedical Science Biological and Marine Sciences Agricultural and Food Sciences Score Engage(d) 0 1 2 3 4 Veterinary Science Mat hematics and Physical Sciences Earth and Environmental Sciences Chemistry and Molecular Biosciences Biomedical Science Biological and Marine Sciences Agricultural and Food Sciences Score Out-of-classroom(f) Figure 1: The Mean and Error Margin of WIL Scores (a-f): WIL Total and Activities by Discipline. There is a pattern in the objective use that reflects a content-focused curriculum The analysis examined which Objectives (Table 1) were most and least frequently used (Figure 2). Courses most frequently included practices that fit Objective 1 (1,182 uses), while Objectives 2, 4, and 5 were moderately used (740, 888, and 766 uses respectively). The least frequently used practices fell into Objective 3 (423 uses) and Objective 6 (153 uses). Objective 6 activities were notably absent in most courses. These results address RQ2. Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 230 Figure 2: The Distribution of Aggregated WIL Activity by Discipline. Objectives and Activities are as shown in Tables 1 and 2. The Pedagogy Group code refers to the Table S1 codes for each pedagogy. The Activity Codes are: Show & Tell (T), Sell (S), Engage (E), Practice (P), and Out-of-classroom (OC). The scores shown are aggregates. For example, the score of 119 for the first row in the Agriculture and Food Science column is an aggregate for all of the Objective 1 Show & Tell pedagogy uses in all of the Agriculture and Food Science courses. Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 231 Discipline is a predictor of WIL activities A multiple linear regression was used to examine how WIL scores vary by class size, year level, and discipline. This analysis determines the relative contributions of the different parameters simul- taneously (Theobald & Freeman, 2014). The research question defines three predictor variables: discipline, year level (both categorical variables) and class size (a continuous variable). The independence of the independent variables was tested using (i) a chi-square test between the categorical variables (discipline and year level) and (ii) an analysis of variance test between each of the categorical variables and the continuous class size variable. There is evidence of strong correlations between all the variables (see Appendix 4, Table S3), with extremely significant correlations (adjusted p value < 10-4) between discipline and class size and year level. Therefore, class size and year were discounted as independent variables, leaving discipline the predictor variable for the analysis. A multiple linear regression was used to test whether course discipline has a significant effect on the WIL score. The results (Table S4) show multiple variables that have significant contributions (asterisked). The null hypothesis is that discipline does not contribute to any of the WIL Activity scores. In Table S4 (see Appendix 4), however, 14 of the regression coefficients have values significantly different from zero, so the null hypothesis is rejected. The predictor variables with significant (positive) contributions according to the model are:  courses within AFS, BIM, EES, and VETS have higher contributions to Engage;  courses within AFS, EES, and VETS have higher contributions to Show & Tell, Sell, and Practice; and  courses within VETS have higher contributions to Out-of-classroom. These results are consistent with the comparisons of average WIL scores (Figure 1) and show, again, that there are differences between the amount and type of WIL offering for students in different disciplines. The results address RQ2 and RQ3. Pairwise regression reveals which disciplines are most similar in their approaches to WIL Figure 2 shows the percentage distribution of WIL Activities within each discipline including the WIL scores for each Activity group. To examine which disciplines were most similar in their pedagogies, pairwise regression was conducted. Figure 3 shows the pairwise regression as a scatter plot matrix (‘pairs plot’) of each discipline area correlated with every other discipline area. Overall, there are strong (R2 > 0.5) and significant (p < 0.05) correlations between each pair of disciplines. Some correlations are weaker than others, but overall, there is general consistency between disciplines in the distribution of WIL Activity scores. The VETS and MPS course groups were the least similar (R2 = 0.55) while the most similar pair are AFS and ESS (R2 = 0.91). The similarities seen in the pairwise regression results are consistent with the WIL score results (Figure 1). These results address RQ3. Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 232 Figure 3: A Scatterplot Matrix of Discipline Areas by Pedagogy Score. The diagonal panels in the plot show the discipline area label and a box plot of the distribution of the pedagogy scores for the discipline area. The upper right panel shows the Pearson’s R squared statistic for each pair of correlations between disciplines, and the lower left shows the scatter plot for each correlation, along with the linear regression line of best fit. Discussion The first aim of this study was to develop a picture of the WIL offerings in a Faculty of Science at a large research-focused public university. It is now evident which disciplines offer WIL more frequently, and which offer significantly fewer WIL experiences. Discipline has a strong impact on teaching (Becher & Trowler, 2001; Mårtensson, Roxå, & Stensaker, 2012), and academics’ teaching practices are most strongly influenced by their close academic peers (Handelsman et al., 2004; Da Silva et al., 2009). Given the breadth of disciplinary teaching in the Faculty and the strongly School-based culture that exists, it was expected that disciplinary differences in WIL practice would be found. The data reveal significant disciplinary differences in the Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 233 amount of WIL offered and the regression analysis shows discipline is a clear predictor of WIL practice. Although published reports (Radloff &Coates, 2010; Edwards et al., 2015; Universities Australia, 2017) indicate that students in some disciplines are offered more placement and project WIL, this is the first time that offerings of all types of WIL have been examined and linked to scientific discipline. This study shows that some disciplines offer more WIL, overall, than others. The VETS, AFS, and EES disciplines have very high average WIL scores, and AFS and EES have the most similar pedagogical approach to WIL. Other disciplines (CMBS, BIM, BIOM, and MBS) have lower scores in overall WIL, and in multiple Activity groups for WIL pedagogies. Maths and Physics (MPS) courses have the lowest overall average WIL score, the lowest average score for each WIL Activity group, and the highest proportion of courses with a zero WIL score. Why do some disciplines offer more WIL than others? This question has been addressed with respect to placement-based WIL. In science, many industry connections that generate WIL placements are also vehicles for research collaborations, and academic staff often see industry connections as a key ‘incentive’ for being involved in WIL activities (Edwards et al., 2015, p. 70). Regional universities also find it relatively easy to place science students with industry because of their applied approach to science curriculum; this contrasts with metropolitan universities which tend to privilege theory and research in the curriculum (Edwards et al., 2015, p. 70). These ideas fit with the disciplinary differences seen in industry-engaged WIL offerings. VETS, AFS, and EES offered more placement WIL than the other disciplines. These three disciplines tend to have a more applied approach to their curriculum, and more of the VETS and AFS courses are offered at our Faculty’s regional campus, where research collaborations with local industry groups are common. In contrast, the other disciplines (MPS, CMBS, BIOM, and BIM) are offered at the UQ city campus. Maths, physics, biochemistry and chemistry fall into the ‘hard pure’ (Becher, 1994) science disciplines, and the academics in these fields are more likely to be theoretical or basic researchers. Hence, their work is likely less amenable to payoff from collaboration with non-university partners who could also interact with students. Another reason for low WIL inclusion in courses may be staff perceptions of the purpose of their teaching and their disciplinary endeavours. Edwards et al. (2015) noted that many science academics felt ‘general ambivalence’ towards WIL development. They suggested some academic staff lack […] any real world experience in their field, which meant expansion into the real world was likely considered too risky (p. 79). There is another possibility, however. One could argue that many academics see their disciplinary material as valuable in an abstract sense and as a vehicle for thinking and research (as opposed to primarily a vehicle for gaining work). Some students may do this too; they may not be asking for explanations of how their learning fits with the world of work. This lack of interest in the practical aspects of gaining employment (both from staff and students) may help explain why there is less WIL overall in some disciplines. In Australia, The Graduate Outcomes Survey (Social Research Centre, 2018) asks students to report their full-time employment status four months after completing their degree. Australian Veterinary Science graduates report full-time employment rates of 81.4%/84.7% (for 2017/2018 respectively), Agriculture and Environmental Studies graduates report 66.3%/68.3%, and Science and Mathematics graduates report 59%/64.6%. It is interesting to compare these broad-brush results with the WIL results obtained for the disciplines in this study. VETS, AFS, and EES all offer high amounts WIL, while the other Sciences, and MPS offer less WIL in their courses. This does not prove causality or a direct relationship between the inclusion of WIL and graduate employment. It may be, however, that the culture of WIL provision (and of normalizing the idea of working in a professional role after graduation) encourages students in Veterinary, Agriculture, and Environmental Science programs to seek employment straight out of their undergraduate degree. In Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 234 contrast, it is also possible that Science and Mathematics students (who get less WIL) are less able to see a future in the workforce and, as a result, they pursue further study or engage in part-time employment that is not directly related to their degree. This idea is consistent with Sin, Reid, and Jones’s (2012) assertion that students who develop a rich understanding of their professional role as an undergraduate will transition more easily to the workforce. Although the volume of WIL differs from one discipline to another, the intent and type of WIL offered in courses studied is similar. The pairwise regression (Figure 3) shows strong and significant correlations between pedagogy use for each discipline, and there is no obvious discipline-related skew visible when the Objective and Activity scores are aligned (Figure 2). All the disciplines most frequently used pedagogies from Objective 1 (To develop [students’] curriculum linked STEM workplace and occupation specific skills and knowledge, and be able to adapt and apply them). In all disciplines, pedagogies from Objectives 3 and 6 (To facilitate [students’] self-understanding) and (To develop [students’] career management skills) were used with low frequency. Objectives 3 and 6 both relate to reflective activities that are not science signature pedagogies (Shulman, 2005). It is possible they are infrequently used because they fit uneasily with the objective, quantitative assessment approach (Yeo & Boman, 2019) used in ‘hard pure’ (Becher, 1994) science courses. It is also possible that the staff in the Faculty do not focus on developing students’ career management skills because they, themselves, are successful academics who have little experience managing a non-university career. The most-commonly-used WIL pedagogies across all the courses (O1T2; O1S1; O1E2; O4T2; O5T1) focus on explicating the link between the course experience and work; teacher modeling and explanation of work practices; or helping students use and recognise professional behaviours and artefacts. Thus, across the courses examined in this study, there is purposeful linkage of curriculum to work and also to professional, discipline-authentic practice that reflects what scientists ‘do’ (Rowland, Pedwell. Lawrie et al., 2016). Interestingly, this linkage is most often enacted through Show & Tell pedagogies, which involve telling students about how the curriculum […] relates to the workplace (Edwards et al., 2015, p. 53). Although Show & Tell pedagogies are easy to implement, ‘telling’ does not engage students in work-related reflection or help them generate new ways of knowing, within and through practice (Orrell, 2011, p. 8). Despite this limitation, the results do indicate that faculty members are interested in relating courses to students’ employment pathways, and that they do this frequently. Since this study was done in a Faculty that was not supporting WIL through a central mechanism, the results are heartening. The second aim was to inform the conversation about WIL in Science more broadly. In particular, the focus was on understanding the intent and type of WIL incorporated in curricula. As noted earlier, literature reports (Radloff & Coates, 2010; Edwards et al., 2015; Universities Australia, 2017) suggest that students in some science disciplines get poor exposure to placement and internship WIL. It is tempting to extrapolate from these data to envisage science WIL as a pedagogical wasteland, and this study questioned whether this is truly the case. The data, however, paint a picture of WIL practice that is multifaceted, and richly considerate of helping students understand how their university education will link to employment. Consistent with published reports, the results show a low proportion of Out-of-classroom WIL (28% of courses). When all potential pedagogies are considered, however, almost all the responding courses in the study (92%) contained some form of WIL; on average, each course uses 15 to 16 WIL pedagogies. This was a surprising high number of pedagogies used per course and also a high number of courses that incorporated WIL. The teaching approaches in these courses provide a way to offer WIL to students in large classes, in disciplines where it may be difficult to source industry placements, and in cohorts where the future careers of the students are unclear (Rowland et al., 2020). Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 235 There are limitations to this study. For several reasons, WIL in courses may be under- or mis- reported. Some instructors did not open the survey link. When telephoned to check on their response intentions, some reported their course had ‘no WIL’, based on their understanding of WIL. These courses were counted as non-responses, but they may have contained some WIL. In addition, only course coordinators were invited to participate, but most courses are team-taught. Course coordinators can be incompletely aware of the Activities, Objectives, and pedagogies (Table S1) used in their courses. As noted earlier, the double-barrelled questions in the survey may also cause under- reporting of WIL. Coordinators of 393 courses were asked for information; 265 responded, 21 reported no WIL, but 128 courses were silent. Thus, it can be said that 62% of the courses examined had some form of WIL, while 38% of courses have no, or unknown amounts of, WIL. Implications for practice and future research Modern universities have increasingly valued research over teaching (Anderson et al., 2011; Blakey, Khachikian, & Lemus, 2017) and WIL, described as the poor cousin of teaching (Edwards et al., 2015, p. 89) has occupied the bottom rung of the value ladder. This lack of recognition has stunted WIL as a signature pedagogy in science, but the disturbing employment statistics for some science graduates suggests a need to re-think the approach to helping students build their sense of themselves as professionals after graduation. Krause (2020) notes the economic and policy imperative to prepare employable graduates has had a noteworthy impact on shaping undergraduate curricula internationally (p. 7). In science education, it is now time for this concern to translate into course-based exposure to employers, the science workplace, and professional practice development. This study shows that the courses in the Faculty of Science incorporate a diverse set of approaches to WIL, but it has also identified disciplines in which to prioritize future WIL development. In the Faculty of Science, Veterinary Science offers high amounts of WIL in all its forms. The disciplines of Agriculture and Food Science and Earth and Environmental Science use similar pedagogical approaches to deliver their rich WIL offerings. There is space, however, to increase all forms of WIL provision for students in the disciplines of Mathematics and Physics, Chemistry and Molecular Biosciences, Biology and Marine Science, and Biomedical Sciences. There is also space to harness the frequently-used pedagogies in the Faculty, and develop them into educational activities and assessments that are more likely to foster active student learning about work. These frequently-used pedagogies are mostly in-class activities, but how effective is in-classroom WIL in helping students learn about the world of work? A full answer to this question is beyond the scope of this paper (and indeed, the answer is unclear and is likely to be different based on discipline, context, and the way in-classroom WIL is enacted). There are several frameworks for assessing the quality of WIL and WIL assessment (e.g., Smith, 2012; Oliver, 2015; Kaider et al., 2017; Campbell et al., 2019). Applying these to the in-classroom WIL pedagogies may help educators make decisions about which in-classroom WIL pedagogies to prioritise for their context. One way to build value in all types of WIL is to increase the amount of reflection that students do. Objective 3 (To facilitate self-understanding) was minimally used in the courses studied, but there is no reason why it cannot be included in all science curricula. Students have different innate capacities to reflect on their employability development (Scott & Willison, 2021), although it is known that reflection creates and develops practitioners capable of demonstrating their progression towards learning outcomes and required standards (Helyer, 2015, p. 15). It is not difficult to change a curriculum to (i) introduce a learning outcome around work- preparedness and (ii) require students reflect on how the curriculum has helped them meet this Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 236 learning outcome. This practice yields two positives. It elevates ‘hidden’ WIL to visible WIL. It also encourages students to recognise the WIL and the learning they build as a result. Over the past three years this has been done in the science courses at UQ, and important student learning has been observed as a result. This project will be reported separately. This study provides the first pan-Faculty evidence of systematic differences in WIL provision and workplace preparation for students in different disciplines of science. In all, the results suggest that undergraduate university students from different disciplines are likely to have different levels of exposure to workplace practices, relevant professionals, and discussions of how their curriculum is relevant to their future workplace. No claims are made around whether this WIL exposure changes students’ employability, but the results serve as a catalyst for reflexive examination of science curricula and transformational change in classroom pedagogical practice around preparing students for work. Acknowledgments We thank the academic staff in the UQ Faculty of Science for their generosity as they answered the survey. Thanks are also due to Rhianna Pedwell and Stephanie Beames, who both assisted in data collection. This work was funded by a UQ Teaching and Learning Fellowship, and the support of the University is appreciated. References AIG (2016). Workforce Development Needs Survey Report. Australian Industry Group. https://www.aigroup.com.au Anderson, W.A., Banerjee, U., Drennan, C.L., Elgin, S.C.R., Epstein, I.R., Handelsman, J., Hatfull, G.F., Losick, R., O’Dowd, D.K., Olivera, B.M., Strobel, S.A., Walker, G.C., & Warner, I.M. (2011). Changing the culture of science education at research universities. Science, 331, 152–153. doi:10.1126/science.1198280. Beard, C., & Wilson, J.P. (2006). Experiential learning: Best practice handbook for educators and trainers (2nd edn). London: Kogan Page. Becher, T. (1994). The significance of disciplinary differences. Studies in Higher Education, 19(2), 151–161. https://doi.org/10.1080/03075079412331382007 Becher, T., & Trowler, P.R. (2001). Academic tribes and territories, 2nd ed. Buckingham: Open University Press. Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society SeriesB, 57(1), 289–300. Blakey, E., Khachikian, C., & Lemus, D. (2017). Increasing research requirements for tenure at teaching universities: Mission creep or mission critical? Teacher-Scholar, 8, Article 3. http://scholars. su.edu/ts/vol8/iss1/3 Campbell, M., Russell, L., Smith, L., McAllister, L., Tunny, R., Thomson, K., & Barrett, M. (2019). A framework for the institutional quality assurance of work integrated learning. Australian Collaborative Education Network. Da Silva, K.B., Fawcett, R., Hunter, N., Buckley, P., Roberts, M., Dent, L., Wood, D., & Gannaway, D. (2009). Raising the profile of teaching and learning: Scientists leading scientists. Sydney: Office for Learning and Teaching. Dean, B. A., & Campbell, M. (2020). Reshaping work-integrated learning in a post-COVID-19 world of work. IJWIL, 21, 356–364. Deloitte Access Economics (2014). Australia’s STEM workforce: A survey of employers. ACT: Deloitte. Edwards, D., Perkins, K., Pearce, J., & Hong, J. (2015). Work integrated learning in STEM in Australian Universities. Canberra: Office of the Chief Scientist Australia. https://bit.ly/31YqwOL Graduate Careers Australia (2015). Graduate outlook 2015. The report of the 2015 graduate outlook survey: Perspectives on graduate recruitment. https://bit.ly/2ZoRY6G Handelsman, J., Ebert-May, D., Beichner, R., Bruns, P., Chang, A., DeHaan, R., Gentile, J., Lauffer, S., Stewart, J., Tilghman, S.M., & Wood, W.B. (2004). Scientific teaching. Science, 304(5670), 521–522. doi:10.1126/science.1096022 https://www.aigroup.com.au/ https://doi.org/10.1080/03075079412331382007 https://bit.ly/31YqwOL https://bit.ly/2ZoRY6G Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 237 Helyer, R. (2015) Learning through reflection: the critical role of reflection in work-based learning (WBL). Journal of Work-Applied Management, 7(1), 15–27. doi:10.1108/JWAM-10-2015-003 Kaider, F., Hains-Wesson, R., & Young, K. (2017). Practical typology of authentic Work-Integrated Learning activities and assessments. Asia-Pacific Journal of Cooperative Education: Special Edition, 18(2), 153– 165. Kay, J., Ferns, S., Russell, L., Smith, J., & Winchester-Seeto, T. (2019) The emerging future: Innovative models of work-integrated learning. International Journal of Work Integrated Learning, 20(4), 410–413. Krause, K.-L. (2020): Vectors of change in higher education curricula. Journal of Curriculum Studies, (online preprint) doi:10.1080/00220272.2020.1764627 Loomes, S., Owens, A., & McCarthy, G. (2019). Patterns of recruitment of academic leaders to Australian universities and implications for the future of higher education. Journal of High Education Policy and Management, 41(2), 137–152. doi:10.1080/1360080X.2019.1565296 Mårtensson, K., Roxå, T., & Stensaker, B. (2012). From quality assurance to quality practices: An investigation of strong microcultures in teaching and learning. Studies in Higher Education, 39(4), 1–12. doi:10.1080/03075079.2012.709493 Norton, A., & Cakitaki, B. (2016). Mapping Australian higher education 2016. Melbourne: Grattan Institute. Nulty, D.D. (2008). The adequacy of response rates to online and paper surveys: What can be done? Assess & Evaluation in Higher Education, 33(3), 301–314. doi:10.1080/02602930701293231 Office of the Chief Scientist (2020). Australia's STEM Workforce. Canberra: Office of the Chief Scientist Australia. Oliver, B. (2015). Redefining graduate employability and work integrated learning: Proposals for effective higher education in disrupted economies. Journal of Teaching and Learning for Graduate Employment, 6(1), 56–65. doi:10.21153/jtlge2015vol6no1art573 Orrell, J. (2011). Good Practice Report: Work-integrated learning. Canberra: Australian Learning and Teaching Council. Patrick, C.-J., Peach, D., Pocknee, C., Webb, F., Fletcher, M., & Pretto, G. (2008). The WIL (Work Integrated Learning) report: A national scoping study. Canberra: Australian Learning and Teaching Council. Radloff, A., & Coates, H. (2010). Doing more for learning: Enhancing engagement and outcomes: Australasian Survey of Student Engagement: Australasian Student Engagement Report. Camberwell: Australian Council for Educational Research (ACER). (https://www.acer.org/au/about-us/corporate-profile) Rowland, S., Gannaway, D., Pedwell, R., Adams, P., Evans, R., Bonner, H., & Wong, K.S. (2020) Legitimising transgression: Design and delivery of a science work integrated learning program that draws on students’ extant work in diverse, non-science fields. Higher Education Research and Development, 39(2), 318–331. doi: 10.1080/07294360.2019.1668364 Rowland, S., Pedwell, R., Lawrie, G., Lovie-Toon, J., & Hung, Y. (2016). Do we need to design course-based undergraduate research experiences for authenticity? CBE Life Sciences Education. 15(4):np doi: 10.1187/cbe.16-02-0102. Scott, F.J., & Willison, D. (2021) Students’ reflections on an employability skills provision. Journal of Further and Higher Education, 1–16. doi:10.1080/0309877X.2021.1928025 Shulman, L.S. (2005). Signature pedagogies in the professions. Daedalus, 134(3), 52–59. Sin, S., Reid, A., & Jones, A. (2012). An exploration of students’ conceptions of accounting work. Accounting Education, 21(4), 323–340. doi:10.1080/09639284.2012.661604 Small, L., Shacklock, K., & Marchant, T. (2018). Employability: A contemporary review for higher education stakeholders. Journal of Vocational Education & Training, 70(1), 148–166. doi:10.1080/13636820.2017.1394355 Smith, C.D. (2012). Evaluating the quality of work-integrated learning curricula: A comprehensive framework. Higher Education Research and Development, 31(2), 247–262. doi:10.1080/07294360.2011.558072 Tehan, D. (2019). Job-ready graduates key driver of performance-based funding for universities. Online at https://bit.ly/2ZBi0Vb The Social Research Centre (2018). 2018 Graduate Outcomes Survey National Report. Canberra. https://bit.ly/2VSPigP Theobald, R., & Freeman, S. (2014). Is it the intervention or the students? Using linear regression to control for student characteristics in undergraduate STEM education research. CBE-Life Sciences Education, 13(1), 41–48. doi:10.1187/cbe-13-07-0136 Universities Australia (2017). Work Integrated Learning in Universities: Final report. https://apo.org.au/sites/default/files/resource-files/2019-04/apo-nid242371.pdf https://doi/ https://doi/ https://www.acer.org/au/about-us/corporate-profile https://doi.org/10.1080/13636820.2017.1394355 https://doi/ https://bit.ly/2ZBi0Vb https://doi/ Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 238 Wellings, P., Black, R., Craven, G., Freshwater, D., & Harding, S. (2019). Performance-based funding for the Commonwealth Grant Scheme. Report for the Minister for Education – June 2019. Commonwealth of Australia. https://bit.ly/321rGZL Winchester-Seeto, T., & Piggott, L. (2020) ‘Workplace’ or workforce: What are we preparing students for? Journal of University Teaching and Learning Practice, 17(4) 2020. Online at https://ro.uow.edu.au/jutlp/vol17/iss4/11 Yeo, M., & Boman, J. (2019). Disciplinary approaches to assessment. Journal of Further and Higher Education, 43(4), 482–493. doi:10.1080/0309877X.2017.1367371 https://bit.ly/321rGZL https://doi/ Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 239 Appendix 1 Questions from the survey with objectives and practices coded Each answer has a ‘Yes’/’No’ response. There are 55 questions. A course if given a score from 0 to 55 by totalling the ‘Yes’ answers. OBJECTIVE 1 Questions In your course you might: (i) aim to develop students' curriculum-linked STEM workplace or occupation-specific skills and knowledge and (ii) provide students with opportunities to adapt and apply these skills and understandings. Examples of ways you might do this are shown below. Please select all the activities that are part of your course (Yes/No radio buttons provided). Q 1. The curriculum includes workplace specific examples and guest lecturers who articulate the link between curriculum and industry (T) Q 2. Students are given explicit examples in lectures and course notes of how the skills they are developing relate to the workplace (T) Q 3. Students go on observational field trips to workplaces (T) Q 4. There is an explicit focus on why concepts, skills and information in the course are important to the workplace and how they may be applied in the workplace (S) Q 5. Students go on field trips that are structured around directed activities - these focus on how skills and knowledge apply to the workplace/occupation (E) Q 6. Students engage in inquiry based learning with explicit activities that link skills and knowledge to the workplace/occupation (E) Q 7. Students complete self-directed work-related case studies and scenarios with problems to solve using skills and knowledge obtained in the course (P) Q 8. Students complete simulations, where they practice work-relevant skills (P) Q 9. Workplaces or industry partners provide real world issues as the stimuli for student projects which are managed, completed and assessed internally (P) Q 10. Students work as active members of university-based teams to solve workplace problems including reflection and debriefing (placement outside the university) (O) Q 11. Students complete industry or work placement-based projects where topic-specific skills and professional knowledge are applied and linked to the curriculum (placement outside of the university) (O) OBJECTIVE 2 Questions In your course, you might aim to build your students' understanding of: (i) the nature of industry, and (ii) the roles of different occupations as they relate to industry. Examples of ways you might do this are shown below. Please select all the activities that are part of your course (Yes/No radio buttons provided). Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 240 Q 12. Industry guests, professional associations, and academics talk about industry, their occupation and responsibilities within that industry (T) Q 13. There is discussion of professional expectations, ethics and protocols within the industry (T) Q 14. Specific graduate destinations are promoted to students (S) Q 15. Students participate in field trips with explicit reflection on employer expectations about professional practice (E) Q 16. Students reflect on what it means to work as a graduate/professional in a particular industry including academia (E) Q 17. Students complete self-directed case studies/scenarios using skills sets and knowledge across platforms involving different occupations in the workplace (P) Q 18. Students complete simulations of complex problems requiring multiple and nested skills for solutions (P) Q 19. Selected students attend short term placements at the university, sector community or government level (placement outside the university) (O) OBJECTIVE 3 Questions In your course, you might aim to facilitate your students' self-understanding. Examples of ways you might do this are shown below. Please select all the activities that are part of your course (Yes/No radio buttons provided). Q 20. Students observe reflective practice in action by professionals (T) Q 21. The curriculum includes explanations of why reflective practice is critical for developing personal and professional understanding and is the basis of reflective learning by professionals (S) Q 22. The curriculum includes explicit exercises to teach and promote deep reflection (E) Q 23. There are opportunities for self-reflective practices and debrief on process (P) Q 24. There are opportunities for self-reflective practices and debrief on process by external professionals (placement outside the university) (O) OBJECTIVE 4 Questions In your course, you might aim to train your students as professionals to enter a specific STEM industry, with this training taking place in accordance with standards of a defined industry. Examples of ways you might do this are shown below. Please select all the activities that are part of your course (Yes/No radio buttons provided). Q 25. The course design and lectures reflect industry input (T) Q 26. Lecturers share their own (extensive) industry experience (T) Q 27. The course builds a sense of belonging to a profession and the identity of the profession (e.g., discipline branding, course uniforms, badging, graphic design) (S) Q 28. The curriculum includes recognition of industry professional bodies (S) Q 29. The curriculum includes an explicit focus on the whys and hows of professional practice (E) Q 30. There is a work orientated applied focus throughout the course with scaffolded opportunities to apply theory in real world situations and consider issues and potential consequences of decisions (E) Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 241 Q 31. Students complete simulations related to a specific STEM industry (P) Q 32. Students have access to university owned/based clinics (P) Q 33. Students interact with skilled mentors (placement outside of the university) (O) Q 34. Students undergo supervision in a workplace (this could include university as the workplace) (O) Q 35. Students receive ongoing feedback from employers (placement outside of the university) (O) Q 36. Students take part in self-reflection and skilled debriefing (placement outside of the university) (O) Q 37. There is extensive placement throughout the course with scaffolded opportunities to try new things, and understand consequences of decisions (placement outside of the university) (O) Q 38. Students participate in year-long projects with a high degree of student autonomy and responsibility (placement outside of the university) (O) OBJECTIVE 5 Questions In your course, you might aim to develop your students': (i) employability and contextualised language, and (ii) literacy and numeracy skills. Examples of ways you might do this are shown below. Please select all the activities that are part of your course (Yes/No radio buttons provided). Q 39. The curriculum includes activities involving recognition of professional behaviours, communication, documentation, WHS, ethics and regulations, management skills, professional workplace hierarchy, group work dynamics (T) Q 40. The curriculum includes explanations of why and how professional language/literacy/behaviour leads to employability (S) Q 41. The curriculum includes case studies with explicit focus on non-technical skills (E) Q 42. Students receive explicit training in techniques and strategies, related to employability, with contextualised language, literacy, and numeracy (E) Q 43. Students participate in simulated activities, related to employability, with contextualised language, literacy, and numeracy (P) Q 44. Students participate in interdependent group work with explicit roles (P) Q 45. Students participate in role playing exercises (P) Q 46. Students participate in industry based placements with minimal preparation, support or feedback (placement outside of the university) (O) Q 47. Students participate in industry based placements with explicit focus on technical and employability skills, with supports such as mentoring and client feedback (placement outside of the university) (O) OBJECTIVE 6 Questions In your course, you might aim to develop your students' career management skills. Examples of ways you might do this are shown below. Please select all the activities that are part of your course (Yes/No radio buttons provided). Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 242 Q 48. Students receive career advice and skills training (e.g., resume writing) (T) Q 49. Students are shown examples of job ads in relevant fields (T) Q 50. Students hear from alumni speakers who focus on how their degree has proven useful or who illustrate an unusual career pathway (S) Q 51. Students attend industry network events (E) Q 52. Students take part in mock interviews involving industry members (P) Q 53. Students fill out formal applications required for industry projects/placements (O) Q 54.Students participate in competitive processes for winning industry placements (O) Q55. Students initiate work placements and projects (O) Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 243 Appendix 2 Criteria for course selection We selected courses satisfying all of the following criteria: 1. Courses for undergraduate students offered at UQ through a Faculty of Science School, in the year from July 2016 through July 2017. 2. Courses offered for students who are progressing in a Faculty of Science-controlled program (as opposed to ‘service’ courses for students completing a non-Science program in another Faculty). 3. Where a course was offered in duplicate or triplicate (i.e., in multiple semesters each year), we chose the semester offering with the largest enrolment. Process of assigning courses to clusters Each course was assigned to a relevant cluster over three iterations. Courses were assigned to a cluster according to the School owner (first), the material presented in the course (second), and the programs in which the course appeared (third). Most courses are included in a cluster that describes the teaching focus in the course’s administering School. Occasionally, courses are listed in ‘foreign’ clusters (e.g., a SAFS Chemistry course is assigned to CMBS). Incomplete or missing data If one of the 55 WIL questions was not answered for a course record, this was regarded as a missing answer. In total, 3.4% of answers were missing across a total of 14,575 questions over 265 courses. Since the rate of missing answers is very low (<5%) imputation was not considered. Response bias Response bias was investigated within categories relating to key questions of interest to ensure there was not any disproportionate non-response. Figure S1 shows response by category within each area (Discipline, Year Level and Class Size), along with a 95% confidence intervals. A two-sided t-test was performed comparing categories of response for each area to detect categories with significantly different response rates (i.e., if the p value was 0.05 or less). No categories for any area were shown to be significantly different from any other category. Linear regression All R code for the research showing the model, tests for independence and model assumptions, and exploratory data analysis is in Appendix 3. For the linear regression, possible outcome variables are the five WIL activity scores. Our model for the linear regression has five relations that predict the WIL scores for the activities. Each relation has the form: (1) where is an error term and the coefficients are found by multiple linear regression. The variable ranges from 1 to 5 for the five WIL activities predicted. The independent category variables AFS, BIM, BIOM, CMBS, EES, and VETS have the value of 1 for courses in these disciplines and zero otherwise. MPS was chosen as the reference category, as it was consistently the lowest scoring. Thus, any significant coefficients reflect increased WIL scores. Table 5 (see Results) shows coefficients (and their standard errors (SE)) for equation (1) calculated by linear regression. The intercept is the predicted value (estimate) of the baseline case (i.e., the mean for MPS). The model for Out-of-classroom was modelled by a Poisson linear regression as the Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 244 response variable indicated a Poisson distribution and the residuals deviated greatly from normal. A two-tailed t test was used to determine whether each coefficient is significantly different from zero. A variable was considered to contribute significantly if the Benjamini and Hochberg (1995) adjusted p value (p-BH) is less than 0.05. Pairwise regression The data set for correlation analysis consisted of: i) an aggregation of WIL scores by question (pedagogy) as the observations; and ii) each discipline as a feature (see an example of the data layout in Table S2). To retain nuance in the analysis, data at the individual pedagogy question level were used. Pairwise regression was conducted to examine the strength of correlation between all pairs of discipline areas. Here, the significance of the linear relationship between pairs was calculated to check how well aligned each discipline pair was in terms of the aggregate WIL score for each pedagogy (question). Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 245 Appendix 3 R code for analysis of data Work Integrated Learning: Input Variables Test for Independence # This code is designed to test if the input variables are independent. # That is, to test the assumption that the input variables are uncorrelated and independent. # # The variables to be tested are: # # * Discipline by Year Level (Categorial x Categorical); # * Discipline by Enrolments (Categorical x Continuous); # * Year Level by Enrolments (Categorical x Continuous). # Packages required library(data.table) # Summary tables library(moments) # Provides more summary statistical options library(ggplot2) # Plot charts # Import data discipline_year <- read.csv("./Data/input data/discipline_year.csv") names(discipline_year) discipline_enrolments <- read.csv("./Data/input data/discipline_enrolments.csv") names(discipline_enrolments) year_enrolments <- read.csv("./Data/input data/year_enrolments.csv") names(year_enrolments) Chi Squared Tests # Test the hypothesis whether the course disciple is independent of the year level in terms of WIL Score at 0.05 significance level. chi_sq_discipline_year <- chisq.test(discipline_year[,c(2:4)]) chi_sq_discipline_year # The p-value is well below the 0.05 significance level we reject the null hypothesis # that the course discipline is independent of the year level. Anova Tests # Assumptions of ANOVA test: # # * The observations are obtained independently and randomly from the population defined by the factor levels; # * The data of each factor level are normally distributed; and # * These normal populations have a common variance. (Levene's test can be used to check this.) # Compute summary statistics by groups - count, mean, sd for discipline: dt <- data.table(discipline_enrolments) dt[,list(count = length(Course.Code), mean=mean(enrolments), sd=sd(enrolments), skew = skewness(enrolments)),by=discipline] # It appears that there could be strong positive skew in all categories with the exception of VETS. # Box plots # ++++++++++++++++++++ # Plot enrolments by discipline boxplot(enrolments~discipline, data = discipline_enrolments, ylab = "Enrolments", xlab = "Discipline") png("./Results/Plots/box_enrolments x discipline.png", width = 792, height = 792); boxplot(enrolments~discipline, data = discipline_enrolments, ylab = "Enrolments", xlab = "Discipline"); dev.off() # It could be that a lot of the skew arises from a small number of outliters # Given the positve skew, it may be effective to log the results to achieve normality. # Normality plots (qq-plots) Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 246 disp <- c("AFS", "BIM", "BIOM", "CMBS", "EES", "MPS", "VETS") # Iterate over all disciplines (logged) for (j in disp){ # export results fname <- paste("Results/Plots/Norm Plots ", j, ".png", sep="") png(fname, width = 792, height = 792); y = paste("discipline_enrolments_", j, sep="") y <- subset(discipline_enrolments, discipline == j) qqnorm(log(y[,3]), xlab = "Normalised enrolments", main = "") qqline(log(y[,3]), lty = 2) title(paste("Q-Q Plot for ", j, " x Enrolments", sep ="")) dev.off() } # Iterate over all disciplines (not-logged) for (j in disp){ # export results fname <- paste("Results/Plots/Norm Plots (no log) ", j, ".png", sep="") png(fname, width = 792, height = 792); y = paste("discipline_enrolments_", j, sep="") y <- subset(discipline_enrolments, discipline == j) qqnorm(y[,3], xlab = "Normalised enrolments", main = "") qqline(y[,3], lty = 2) title(paste("Q-Q Plot for ", j, " x Enrolments", sep ="")) dev.off() # Close the file on each loop so the next file can open } # It looks as though all discipline categories are positively skewed except for VETS. # # The log transformation appears to create a normal qq-plot in all disciplines other than VETS where it is not effected. # # Hence, the enrolments for disciplines will require a log transformation to normality so as to meet the requirements of ANOVA # Create a new variable under a log transformation discipline_enrolments$log_enrolments <- log(discipline_enrolments[,3]+1) # Compute summary statistics by groups - count, mean, sd for year level: dt <- data.table(year_enrolments) dt[,list(count = length(Course.Code), mean=mean(enrolments), sd=sd(enrolments), skew = skewness(enrolments)),by=Year.Level] # Box plots # ++++++++++++++++++++ # Plot enrolments by Year boxplot(enrolments~Year.Level, data = year_enrolments, ylab = "Enrolments", xlab = "Year Level") png("./Results/Plots/box_enrolments x year.png", width = 792, height = 792); boxplot(enrolments~Year.Level, data = year_enrolments, ylab = "Enrolments", xlab = "Year Level"); dev.off() # Similar to discipline, it could be that a lot of the skew arises from a small number of outliters # Given the positve skew, it may be effective to log the results to achieve normality. yr <- c("Year 1", "Year 2", "Year 3") # Iterate over all disciplines (logged) for (j in yr){ # export results fname <- paste("Results/Plots/Norm Plots ", j, ".png", sep="") Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 247 png(fname, width = 792, height = 792); y = paste("year_enrolments_", j, sep="") y <- subset(year_enrolments, Year.Level == j) qqnorm(log(y[,3]), xlab = "Normalised enrolments", main = "") qqline(log(y[,3]), lty = 2) title(paste("Q-Q Plot for ", j, " x Enrolments", sep ="")) dev.off() } # Iterate over all disciplines (logged) for (j in yr){ # export results fname <- paste("Results/Plots/Norm Plots (no log)", j, ".png", sep="") png(fname, width = 792, height = 792); y = paste("year_enrolments_", j, sep="") y <- subset(year_enrolments, Year.Level == j) qqnorm(y[,3], xlab = "Normalised enrolments", main = "") qqline(y[,3], lty = 2) title(paste("Q-Q Plot for ", j, " x Enrolments", sep ="")) dev.off() } # The qq-plots show a need to log transform prior to ANOVA. # Create a new variable under a log transformation year_enrolments$log_enrolments <- log(year_enrolments[,3]+1) #### Compute one-way ANOVA test # We want to know if there is any significant difference between disciplines in the average enrolments in the. # Compute the analysis of variance discipline_res_aov <- aov(log_enrolments ~ discipline, data = discipline_enrolments) # Summary of the analysis summary(discipline_res_aov) # In this case the ANOVA has a p < 0.05 showing that the mean enrolments between disciplines are different # or that is to say the variance between disciplines is larger than the variance within disciplines. year_res_aov <- aov(log_enrolments ~ Year.Level, data = year_enrolments) # Summary of the analysis summary(year_res_aov) # In this case the ANOVA has a p < 0.05 showing that the mean enrolments between year levels are different # or that is to say the variance between year levels is larger than the variance within year levels. Pairs_Panels # Put p-values in the panels panel.pval <- function(x, y, digits=2, prefix="", cex.cor, ...) { usr <- par("usr"); on.exit(par(usr)) par(usr = c(0, 1, 0, 1)) test <- cor.test(x,y) pval <- test$p.value txt <- format(c(pval, 0.123456789), digits=digits)[1] txt <- paste(prefix, txt, sep="") if(missing(cex.cor)) cex.cor <- 0.8/strwidth(txt) text(0.5, 0.5, txt) } # Put linear regression lines with confidence interval in panels panel.lin<- function(x, y) { points(x,y, pch=21, bg=par("bg"), col = "black",cex=1) #points(x,y, pch=19, col=c("red", "green3", "blue")[groups]) lm.out <- lm(y ~ x) newx = seq(min(x),max(x),by = 0.05) Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 248 conf_interval <- predict(lm.out, newdata=data.frame(x=newx), interval="confidence", level = 0.95) abline(lm.out, col="lightblue") lines(newx, conf_interval[,2], col="red", lty=2) lines(newx, conf_interval[,3], col="red", lty=2) } ## put histograms on the diagonal panel.hist <- function(x, ...) { usr <- par("usr"); on.exit(par(usr)) par(usr = c(usr[1:2], 0, 1.5) ) h <- hist(x, plot = FALSE) breaks <- h$breaks; nB <- length(breaks) y <- h$counts; y <- y/max(y) rect(breaks[-nB], 0, breaks[-1], y, col="cyan", ...) } ## put box plots on the diagonal panel.bxp <- function(x, ...) { usr <- par("usr"); on.exit(par(usr)) par(usr = c(-1.5, 2, usr[3:4])) boxplot(x, add=TRUE) } ## put the r squared value on the upper panels, ## with size proportional to the correlations. panel.corsq <- function(x, y, digits=2, prefix="", cex.cor, ...) { usr <- par("usr"); on.exit(par(usr)) par(usr = c(0, 1, 0, 1)) rsq <- abs(cor(x, y))^2 txt <- format(c(rsq, 0.123456789), digits=digits)[1] txt <- paste(prefix, txt, sep="") if(missing(cex.cor)) cex.cor <- 0.8/strwidth(txt) text(0.5, 0.5, txt, cex = cex.cor * rsq) #print(rsq) } Scatter plot matrices and cluster analysis ### This code produces scatter plot matricies and cluster analysis for WIL data # Import required packages library(ggplot2) # Data visualization library(lattice) source('./Code/pairs_panels.r') # import script for pairs # This dataset has individual aggregated question scores as observations and disciplines as features wil <- read.csv("./Data/input data/wil_scatter.csv") names(wil) summary(wil) # Shows each discipline correllated with each other discipline pairs(wil[,2:8],lower.panel=panel.lin, upper.panel=panel.corsq, diag.panel=panel.bxp, cex.labels=1.5, gap = 0.5, label.pos = 0.8) # Exports the pairs plot to a .png png("./Results/Plots/wil_pair_corsq.png", width = 792, height = 792); pairs(wil[,2:8],lower.panel=panel.lin, upper.panel=panel.corsq, diag.panel=panel.bxp, cex.labels=1.5, gap = 0.5, label.pos = 0.8); dev.off() ## Each discipline shows a strong (R^2>=0.5) and significant (p<=0.05) correllation with every other discipline. Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 249 ## This indicates that across all disciplines the spread of WIL activities is consitent. ## However, some correlations are weaker than others. As one example, MPS shows the weakest correllations ## with VETS and ESS with an r^2 of 0.61 and 0.55 respectively. ## This leads to the question, in terms of activity spread, which disciplines are more similar to each other, ## and which disciplines are more dissimilar. ## To answer this question, clustering disciplines on the basis of activity spread could be insightful. ## Two versions of clustering algorithm were applied: 1) hierarchical clustering and 2) K-means clustering. # The hierarchical clustering algorithm finds clusters as follows: # 1) Compute the distance matrix of the data set using some distance measure (let each data point be a cluster); # 2) Merge the two closets clusters; 3) Update the distance matrix; # 4) Repeat steps 2 and 3 until only a single cluster remains. # Here, the distance matrix is calculated using dist(x, method), where x is the data frame and the default value for method is “euclidean” # (other methods for example include, “manhattan” and “maximum”). # The hierarchical clustering algorithm is performed with the hclust(dist(x, method), method) where the default method is "complete" (max). # However, other methods include "average", "ward.D", "ward.D2", "single" (min), "mcquitty", "median" and "centroid". # The general K-Means algorithm finds clusters as follows: # Given a choice of K, the number of clusters: Select K points as initial centroids randomly; # Repeat: Form K clusters by assigning each point to its closest centroid; # Re-compute the centroids (mean point) of each cluster, until convergence criterion is satisfied. # K-Means clustering was performed using the kmeans(x, k) function, where x is a data frame and k is the number of chosen clusters. # The function outputs were assigned to an object called “cluster”. # This object contains: “cluster”, a vector of integers (1 to k) indicating the cluster to which each point is allocated; # “centers”, a vector containing the cluster centroids; withinss: a vector of within-cluster SSE, with one entry for each cluster; # “tot.withinss”, the sum of withinss; and “Size”, number of points in each cluster. # The following scaling function expresses all values in the column as a proportion. # The data frame is then transposed to relate by discipline. df <- t(apply(wil[,2:8], MARGIN = 2, FUN = function(X) X/sum(X))) disciplines <- rownames(df) # A range of different methods for hclust() were ran and compared to kmeans. # The best agreement between kmeans and hclust was for k=4 for the "ward.D" and ward.D2 method in hclust(). # Ward.D2 however is the correct method of implementing the "Ward" algorithm [1]. meth <- c("complete", "average", "ward.D2", "single", "mcquitty", "median", "centroid") # Iterate over all methods for (j in meth){ fname <- paste("./Results/Plots/hclust", j, ".png", sep="") png(fname, width = 600, height = 796); hc <- hclust(dist(df), method = j) par(mfrow=c(2,2)) for (i in 2:5) { plot(hc, hang = -1, labels = disciplines, main=paste("k=", i, sep ="")) rect.hclust(hc, k=i) } # Close the file on each loop so the next file can open dev.off() } ## Cluster the data into clusters k = 2, 3, 4 and 5. ## using K-Means clustering. Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 250 Plot SSE vs K # Set up a null data frame to store each k and the within # cluster sum of squared error (SSE) within_sse <- data.frame(matrix(NA, nrow = 4, ncol = 2)) colnames(within_sse) <- c("K", "SSE") clust_out <- data.frame(matrix(NA, nrow = 7, ncol = 4)) rownames(clust_out) <- c("AFS", "BIM", "BIOM", "CMBS", "EES", "MPS", "VETS") colnames(clust_out) <- c("k=2", "k=3", "k=4", "k=5") for (k in 2:5) { cluster <- kmeans(df,k) # Store each k in the data frame within_sse[k-1, 1] <- k # Retrieve and store SSE for different values of k in the data frame within_sse[k-1, 2] <- cluster$tot.withinss # Store clusters for each discipline in k clust_out[k-1] <- cluster$cluster } write.csv(clust_out, "./Results/Tables/kmeans clusters.csv") png("./Results/Plots/sse_k.jpg", width = 600, height = 600); ggplot(within_sse, aes(x=K, y=SSE)) + geom_line() + ggtitle("Total SSE wrt Number of Clusters") + theme(plot.title = element_text(hjust = 0.5)); dev.off() # References: # [1] Fionn Murtagh and Pierre Legendre, (2014), Ward’s Hierarchical Agglomerative Clustering Method: Which Algorithms Implement Ward’s Criterion?, Journal of Classification 31:274-295 “Engage Model" # output: html_notebook # --- # Check model data summary summary(wil_model_engage) # Checking for normality in the Engage WIL score. library(car) hist(wil_model_engage$score, main = "Histogram of Engage WIL score", xlab = "WIL score") qqPlot(wil_model_engage$score, main = "Q-Q Plot of Engage WIL score", ylab = "WIL score data quantities", xlab = "WIL score theoretical quantities") # The histogram above indicates a small positive skew and but the q-q plot indicates a reasonably normal distribution. A linear model is therefore applicable. # The procedure is to leave one dummy variable for discipline out as reference. # Although this procedure was run for all combination, SMP had the lowest average # for all disciplines so this will be left out in the final results. wil_engage <- lm(score ~ factor(AFS) + factor(BIM) + factor(BIOM) + factor(CMBS) + factor(EES) + factor(VETS), data = wil_model_engage) summary(wil_engage) wil_engage_sum <- summary(wil_engage) wil_engage_est<-summary(wil_engage)$coefficients[,c(1,4)] wil_engage_se<-summary(wil_engage)$coefficients[,2] wil_engage_ci<-confint(wil_engage) wil_engage_sumtab <- data.frame(wil_engage_est, wil_engage_ci, wil_engage_se) colnames(wil_engage_sumtab) <- c("Estimate", "p-value", "LCL", "UCL", "SE") wil_engage_sumtab <- wil_engage_sumtab[c("Estimate", "LCL", "UCL", "p-value", "SE")] wil_engage_sumtab write.csv(wil_engage_sumtab, "./Data/output data/wil_engage_sumtab_MPS.csv") par(mfrow=c(2,2)) plot(wil_engage) dev.off() Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 251 # A q-q plot of the residuals shows evidence of normally distributed residuals. "Outside of classroom Model" # output: html_notebook # --- # Check model data summary summary(wil_model_osc) # Checking for normality in the Outside of classroom WIL score. library(car) hist(wil_model_osc$score, main = "Histogram of Outside of classroom WIL score", xlab = "WIL score") qqPlot(wil_model_osc$score, main = "Q-Q Plot of Outside of classroom WIL score", ylab = "WIL score data quantities", xlab = "WIL score theoretical quantities") # The histogram above indicates a positive skew and the q-q plot indicates a non-normal distribution. # A linear model is therefore not applicable here. # WIL scores are discrete values and could be viewed as counts. Hence, a Poisson model is applied. # The procedure is to leave one dummy variable for discipline out as reference. # Although this procedure was run for all combination, SMP had the lowest average # for all disciplines so this will be left out in the final results. wil_osc <- glm(score ~ factor(AFS) + factor(BIM) + factor(BIOM) + factor(CMBS) + factor(EES) + factor(VETS), family = quasipoisson(), data = wil_model_osc) summary(wil_osc) # A poisson model relies on the assumption that the mean and variance are equal. Hence, the model does not allow for overdispersion. # An indicator of overdispersion is the relationship between the Residual deviance and the degrees of freedom. # The Residual deviance and the degrees of freedom should be similar or at least the Residual deviance should not be much larger than the degrees of freedom. # In the model above, the Residual deviance is three times larger than the degrees of freedom. # Hence, a quasipoisson is used instead which allows an estimation of the model parameters without fully knowing the error distribution of the response variable [1]. # wil_osc_sum <- summary(wil_osc) # To transform the estimates and confidence intervals into non-log values, they needed transforming. # The model coefficients represent the log of the mean value for the variable (V) minus the log of the mean value for the reference variable (R) (the intercept in this case). # Hence, to transform the log values into the non-log values say: # x = ln(V) - ln(R) # exp(x) = exp(ln(V/R)) # exp(x) = V/R # V = exp(x)*R # Therefore the difference between V and R is: # exp(x)*R - R wil_osc_est<-c(exp(summary(wil_osc)$coefficients[1,1]), exp(summary(wil_osc)$coefficients[2:7,1])*exp(summary(wil_osc)$coefficients[1,1]) - exp(summary(wil_osc)$coefficients[1,1])) wil_osc_p<-summary(wil_osc)$coefficients[,4] wil_osc_lcl <- c(exp(confint(wil_osc)[1,1]), exp(confint(wil_osc)[2:7,1])*exp(confint(wil_osc)[1,1]) - exp(confint(wil_osc)[1,1])) wil_osc_ucl <- c(exp(confint(wil_osc)[1,2]), exp(confint(wil_osc)[2:7,2])*exp(confint(wil_osc)[1,2]) - exp(confint(wil_osc)[1,2])) wil_osc_se<-exp(summary(wil_osc)$coefficients[,2]) wil_osc_sumtab <- data.frame(wil_osc_est, wil_osc_lcl, wil_osc_ucl, wil_osc_p, wil_osc_se) colnames(wil_osc_sumtab) <- c("Estimate", "LCL", "UCL", "p-value", "SE") wil_osc_sumtab write.csv(wil_osc_sumtab, "./Data/output data/wil_osc_sumtab_MPS.csv") # Model fit: Pseudo R squared wil_osc_prs <- 1 - wil_osc$deviance/wil_osc$null.deviance wil_osc_prs Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 252 par(mfrow=c(2,2)) plot(wil_osc) dev.off() # References: # [1] McCullagh, P. and Nedler, J. A. (1989), Generalised Linear Models, London, Chapman and Hall/CRC. "Practice" # output: html_notebook # --- # Check model data summary summary(wil_model_practice) # Checking for normality in the Practice WIL score. library(car) hist(wil_model_practice$score, main = "Histogram of Practice WIL score", xlab = "WIL score") qqPlot(wil_model_practice$score, main = "Q-Q Plot of Practice WIL score", ylab = "WIL score data quantities", xlab = "WIL score theoretical quantities") # The histogram above indicates a small positive skew and but the q-q plot indicates a reasonably normal distribution. A linear model is therefore applicable. # The procedure is to leave one dummy variable for discipline out as reference. # Although this procedure was run for all combination, SMP had the lowest average # for all disciplines so this will be left out in the final results. wil_practice <- lm(score ~ factor(AFS) + factor(BIM) + factor(BIOM) + factor(CMBS) + factor(EES) + factor(VETS), data = wil_model_practice) summary(wil_practice) wil_practice_sum <- summary(wil_practice) wil_practice_est<-summary(wil_practice)$coefficients[,c(1,4)] wil_practice_se<-summary(wil_practice)$coefficients[,2] wil_practice_ci<-confint(wil_practice) wil_practice_sumtab <- data.frame(wil_practice_est, wil_practice_ci, wil_practice_se) colnames(wil_practice_sumtab) <- c("Estimate", "p-value", "LCL", "UCL", "SE") wil_practice_sumtab <- wil_practice_sumtab[c("Estimate", "LCL", "UCL", "p-value", "SE")] wil_practice_sumtab write.csv(wil_practice_sumtab, "./Data/output data/wil_practice_sumtab_MPS.csv") par(mfrow=c(2,2)) plot(wil_practice) dev.off() #A q-q plot of the residuals shows evidence of normally distributed residuals. "Sell" # output: html_notebook # --- # Check model data summary summary(wil_model_sell) # Checking for normality in the Sell WIL score. library(car) hist(wil_model_sell$score, main = "Histogram of Sell score", xlab = "WIL score") qqPlot(wil_model_sell$score, main = "Q-Q Plot of Sell WIL score", ylab = "WIL score data quantities", xlab = "WIL score theoretical quantities") # The histogram above indicates a small positive skew and but the q-q plot indicates a reasonably normal distribution. A linear model is therefore applicable. # The procedure is to leave one dummy variable for discipline out as reference. # Although this procedure was run for all combination, SMP had the lowest average # for all disciplines so this will be left out in the final results. wil_sell <- lm(score ~ factor(AFS) + factor(BIM) + factor(BIOM) + factor(CMBS) + factor(EES) + factor(VETS), data = wil_model_sell) summary(wil_sell) wil_sell_sum <- summary(wil_sell) wil_sell_est<-summary(wil_sell)$coefficients[,c(1,4)] Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 253 wil_sell_ci<-confint(wil_sell) wil_sell_se<-summary(wil_sell)$coefficients[,2] wil_sell_sumtab <- data.frame(wil_sell_est, wil_sell_ci, wil_sell_se) colnames(wil_sell_sumtab) <- c("Estimate", "p-value", "LCL", "UCL", "SE") wil_sell_sumtab <- wil_sell_sumtab[c("Estimate", "LCL", "UCL", "p-value", "SE")] wil_sell_sumtab write.csv(wil_sell_sumtab, "./Data/output data/wil_sell_sumtab_MPS.csv") par(mfrow=c(2,2)) plot(wil_sell) dev.off() # A q-q plot of the residuals shows evidence of normally distributed residuals. "Show & Tell" # output: html_notebook # --- # Check model data summary summary(wil_model_st) # Checking for normality in the Show and Tell WIL score. library(car) hist(wil_model_st$score, main = "Histogram of Show and Tell score", xlab = "WIL score") qqPlot(wil_model_st$score, main = "Q-Q Plot of Show and Tell WIL score", ylab = "WIL score data quantities", xlab = "WIL score theoretical quantities") # The distribution apart from a strong peak in the first bin appears more uniform. The q-q plot shows some deviation at either end of the plot so there is a small amount of deviation from normal. # The procedure is to leave one dummy variable for discipline out as reference. # Although this procedure was run for all combination, SMP had the lowest average # for all disciplines so this will be left out in the final results. wil_st <- lm(score ~ factor(AFS) + factor(BIM) + factor(BIOM) + factor(CMBS) + factor(EES) + factor(VETS), data = wil_model_st) summary(wil_st) wil_st_sum <- summary(wil_st) wil_st_est<-summary(wil_st)$coefficients[,c(1,4)] wil_st_ci<-confint(wil_st) wil_st_se<-summary(wil_st)$coefficients[,2] wil_st_sumtab <- data.frame(wil_st_est, wil_st_ci, wil_st_se) colnames(wil_st_sumtab) <- c("Estimate", "p-value", "LCL", "UCL", "SE") wil_st_sumtab <- wil_st_sumtab[c("Estimate", "LCL", "UCL", "p-value", "SE")] wil_st_sumtab write.csv(wil_st_sumtab, "./Data/output data/wil_st_sumtab_MPS.csv") par(mfrow=c(2,2)) plot(wil_st) dev.off() # A q-q plot of the residuals shows evidence of normally distributed residuals. "Work Integrated Learning: Regression Model" # output: html_notebook # --- # This code prepares all datasets for modelling each activity. # Import data wil_model <- read.csv("./Data/input data/wil_model.csv") # Check data imported without anomalies names(wil_model) summary(wil_model) # Create subsets for each activity. wil_model_engage <- subset(wil_model, activity == "Engage") wil_model_osc <- subset(wil_model, activity == "Outside of classroom") wil_model_practice <- subset(wil_model, activity == "Practice") wil_model_sell <- subset(wil_model, activity == "Sell") Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 254 wil_model_st <- subset(wil_model, activity == "Show & Tell") ### Linear regression has five key assumptions: # * Linear relationship: Linear regression needs the relationship between the independent and dependent variables to be linear. # In this case, the dependent is continuous and the independent is categorical (i.e. 0 or 1). The main issue is to itentify # that the errors are normally distributed in the linear relationship between the two. Hence test with a univariate logistic regression and inspect the residuals for normality. # * Multivariate normality: The linear regression analysis requires all variables to be multivariate normal. # This assumption can best be checked with a histogram or a Q-Q-Plot. Normality can be checked with a goodness of fit test, e.g., the Kolmogorov-Smirnov test. When the data is not normally distributed a non-linear transformation (e.g., log-transformation) might fix this issue. # * No or little multicollinearity: Multicollinearity occurs when the independent variables are too highly correlated with each other. # Multicollinearity may be tested with three central criteria: # 1) Correlation matrix - when computing the matrix of Pearson's Bivariate Correlation among all independent variables the correlation coefficients need to be smaller than 1. # 2) Tolerance - the tolerance measures the influence of one independent variable on all other independent variables; the tolerance is calculated with an initial linear regression analysis. Tolerance is defined as T = 1 - R? for these first step regression analysis. With T < 0.1 there might be multicollinearity in the data and with T < 0.01 there certainly is. # 3) Variance Inflation Factor (VIF) - the variance inflation factor of the linear regression is defined as VIF = 1/T. With VIF > 10 there is an indication that multicollinearity may be present; with VIF > 100 there is certainly multicollinearity among the variables. # If multicollinearity is found in the data, centering the data (that is deducting the mean of the variable from each score) might help to solve the problem. However, the simplest way to address the problem is to remove independent variables with high VIF values. # Fourth, linear regression analysis requires that there is little or no autocorrelation in the data. Autocorrelation occurs when the residuals are not independent from each other. For instance, this typically occurs in stock prices, where the price is not independent from the previous price. # * No auto-correlation: Autocorrelation occurs when the residuals are not independent from each other. you can test the linear regression model for autocorrelation with the Durbin-Watson test. Durbin-Watson's d tests the null hypothesis that the residuals are not linearly auto-correlated. # * Homoscedasticity: The last assumption of the linear regression analysis is homoscedasticity. The scatter plot is good way to check whether the data are homoscedastic (meaning the residuals are equal across the regression line). # See separate modules for models on each WIL activity. # After all models are run and outputs are complied, run the code below. # The model results are based on the MPS discipline as the base reference. # A number of coefficients in the model are significant and so to control for the # false discovery rate over multiple tests by the Benjamini and Hochberg test is applied: # All p value data has been compiled in a single file: wil_model_pvalues <- read.csv("./Data/input data/lm_results.csv") # Check data names(wil_model_pvalues) summary(wil_model_pvalues) # Create a subset with MPS as the reference value wil_model_pvalues_mps <- subset(wil_model_pvalues, discipline.reference == "MPS" & discipline.comparison != "MPS (Intercept)") p.value.bh <- round(p.adjust(wil_model_pvalues_mps$p.value, "BH"), 4) wil_model_coeffs_pvals_mps <- data.frame(wil_model_pvalues_mps, p.value.bh) wil_model_coeffs_pvals_mps <- wil_model_coeffs_pvals_mps[order(p.value.bh),] wil_model_coeffs_pvals_mps View(wil_model_coeffs_pvals_mps) write.csv(wil_model_coeffs_pvals_mps, "./Data/output data/bh_mps_results.csv") Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 255 # The null hypothesis is that none of the predictor variables (in this case the course discipline), contribute to any of the WIL activities. # In fact 14 out of the 30 coefficients have values significantly different from zero difference between the coefficient and the reference value. # The following contributions are: # # * Courses within Earth and Environmental Sciences (EES), VETSs (VETS), # Agriculture and Food Sciences (AFS) and Biological and Marine Sciences (BIM) # disciplines have higher contributions to Engage (i.e. Field trips structured around directed activities that focus on how skills and knowledge apply to the workplace/occupation). # # * Courses within Earth and Environmental Sciences (EES), VETSs (VETS) and Agriculture and Food Sciences (AFS) disciplines # have higher contributions to: # + Practice (i.e. Self-directed case studies and scenarios with problems to solve using skills and knowledge obtained in course); # + Sell (i.e. Explicit focus on why concepts, skills and information are important to the workplace and how they may be applied in the workplace); and # + Show and Tell (i.e. Workplace specific examples and guest lecturers who articulate the link between curriculum and industry). # # * Courses within the VETSs (VETS) discipline have higher contributions to Outside of classroom # (i.e. Students as active members of university based teams to solve workplace problems including reflection and debriefing). Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 256 Appendix 4 Table S1: The WIL Objectives, Activities, and Pedagogies Matrix used to develop the Survey (modified from Edwards et al., 2015) Objectives of WIL Specific Activities to achieve the Objectives Show & Tell (T) Sell (S) Engage (E) Practice (P) Academic, industry and community settings (OC) In-classroom Pedagogies1 Out-of-classroom Pedagogies G e n e ri c S ci e n ce W IL 1. To develop curriculum linked STEM workplace & occupation specific skills & knowledge, & be able to adapt & apply them O1T1: Work-specific examples & guest lecturers who articulate the link between curriculum & industry O1T2: Explicit examples in lectures & course notes of skills relating to work O1T3: Observational field trips to workplaces (O1T3) O1S1: Explicit focus on why concepts, skills & information are important to work & how they may be applied during work O1E1: Field trips structured around activities that focus on how skills & knowledge apply to work O1E2: Inquiry-based learning with explicit activities linking skills & knowledge to work O1P1: Self-directed work- related case studies & scenarios with problems to solve using knowledge & skills from the course O1P2: Simulations, where students practice work-relevant skills O1P3: Workplace provides real world issues for use in student projects, which are conducted & assessed internally O1OC1: Students as members of university- based teams that solve workplace problems; including reflection & debriefing O1OC2: Industry or work placement-based projects where topic-specific skills & professional knowledge are applied & linked to the curriculum 2. To build an understanding of the nature of industry & the roles of different occupations as they relate to industry O2T1: Industry guests, professional associations, & academics talk about industry, their occupation & responsibilities within that industry O2T2: Discussion of professional expectations, ethics & protocols within the industry O2S1: Promotion of specific graduate destinations O2E1: Field trips with explicit reflection on employer expectations about professional practice O2E2: Reflection on what it means to work as a graduate & professional in a particular industry O2P1: Self-directed case studies & scenarios using skills sets & knowledge across platforms involving different occupations in the workplace O2P2: Simulations of complex problems requiring multiple & nested skills for solutions O2OC1: Selected students for short term placement at the university, sector community or government level 3. To facilitate self- understanding O3T1: Observation of reflective practice in action by professionals O3S1: Explain why reflection is critical for developing personal & professional understanding O3E1: Explicit exercises to teach & promote deep reflection O3P1: Opportunities for self- reflective practices & debrief on process O3OC1: Opportunities for self-reflective practices & debrief on process by external professionals Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 257 In d u st ry s p e ci fi c W IL 4. To train professionals to enter a specific STEM industry in accordance with the standards of a defined industry O4T1: Course design & lectures reflect industry input O4T2: Lectures share their own industry experience O4S1: Build a sense of belonging to a profession & professional identity O4S2: Recognition of industry professional bodies O4E1: Explicit focus on the why & how of professional practice O4E2: Work-orientated, applied focus throughout course; scaffolded opportunities to apply theory, consider issues, & discuss consequences of decisions O4P1: Simulations related to a STEM-specific industry O4P2: University owned or based clinics O4OC1: Skilled mentors O4OC2: Supervision in workplace (including university workplaces) O4OC3: Ongoing feedback from employers O4OC4: Self-reflection & skilled debriefing O4OC5: Extensive placement throughout course with scaffolded opportunities to try new things & understand consequences of decisions O4OC6: Year-long projects with significant student autonomy & responsibility E m p lo y a b il it y & c a re e rs 5. To develop employability & contextualized language, literacy & numeracy skills O5T1: Recognition & use of professional behaviors & artefacts O5S1: Explain why & how professional language, literacy, & behavior leads to employability O5E1: Case studies with explicit focus on non- technical skills O5E2: Explicit training in techniques & strategies O5P1: Simulated activities relevant to employability O5P2: Interdependent group work with explicit roles O5P3: Role playing O5OC1: Industry placements with minimal preparation, support or feedback O5OC2: Mentored, feedback-rich industry placements with explicit focus on technical & employability skills 6. To develop career management skills O6T1: Career advice & skills training O6T2: Examples of job ads in relevant fields O6S1: Alumni speakers who (i) focus on how their course was useful or (ii) illustrate an unusual career path O6E1: Industry network events O6P1: Mock interviews involving industry members O6OC1: Formal applications required for industry projects & placements O6OC2: Competitive processes for winning industry placements O6OC3: Student-initiated work placements & projects 1. Each pedagogy has a code that identifies its Objective and Activity group, as well as its number in the pedagogy list (e.g., code O1T1 denotes the pedagogy as the first “Show & Tell” Activity in the Objective 1 group; code O5P3 denotes the pedagogy as the third “Practice” Activity in the Objective 5 group. Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 258 Table S2: Total and Activity WIL Scores for Different Subsamples Category N WIL Total Show & Tell Sell Engage Practice Outside of classroom M (SD) M (SD) M (SD) M (SD) M (SD) M (SD) All categories 265 15.71 (10.58) 4.58 (3.05) 2.54 (1.91) 3.76 (2.58) 3.93 (2.93) 0.90 (2.41) Year Level First 44 14.89 (11.01) 4.55 (3.08) 2.84 (2.05) 3.59 (2.38) 3.14 (2.75) 0.77 (2.31) Second 71 12.56 (8.94) 3.82 (2.95) 2.08 (1.79) 3.17 (2.50) 3.11 (2.78) 0.39 (1.13) Third 150 17.44 (10.85) 4.95 (3.03) 2.67 (1.91) 4.09 (2.63) 4.55 (2.93) 1.17 (2.82) Class Size Less than 50 124 17.61 (11.16) 5.09 (3.16) 2.70 (1.92) 4.18 (2.78) 4.35 (3.01) 1.29 (2.99) 50-99 57 16.14 (9.04) 4.63 (2.83) 2.74 (1.93) 4.00 (2.43) 4.40 (2.92) 0.38 (0.75) 100-200 47 15.36 (10.55) 4.55 (2.83) 2.57 (1.99) 3.47 (2.28) 3.87 (2.72) 0.89 (2.43) Over 200 37 9.11 (8.20) 2.84 (2.67) 1.68 (1.56) 2.35 (1.92) 1.86 (2.03) 0.38 (1.50) Discipline Agricultural and Food Sciences 60 17.93 (10.38) 5.67 (2.65) 2.97 (1.85) 4.07 (2.68) 4.18 (3.28) 1.07 (2.51) Biological and Marine Sciences 30 13.37 (7.68) 3.67 (2.47) 2.27 (1.46) 3.57 (2.19) 3.30 (2.25) 0.57 (1.14) Biomedical Science 14 12.14 (7.09) 3.86 (2.85) 1.79 (1.37) 3.14 (1.99) 2.86 (2.07) 0.50 (1.09) Chemistry and Molecular Biosciences 37 12.51 (9.39) 3.46 (2.99) 1.84 (1.89) 3.08 (2.33) 3.49 (2.65) 0.65 (1.74) Earth and Environmental Sciences 58 19.14 (10.34) 5.53 (2.93) 3.05 (1.94) 4.72 (2.54) 4.86 (2.94) 0.97 (2.86) Mathematics and Physical Sciences 43 9.19 (9.60) 2.49 (2.72) 1.65 (1.76) 2.21 (2.30) 2.37 (2.40) 0.47 (2.18) Veterinary Science 23 23.83 (10.94) 6.70 (2.48) 3.78 (1.91) 5.13 (2.36) 6.04 (2.64) 2.17 (3.59) Note: For each category, the mean scores are given first with the standard deviations in parentheses. Rowland, S., & Blundell, D. (2021). Discipline predicts Work Integrated Learning (WIL) practice in Science courses. Journal of Teaching and Learning for Graduate Employability, 12(2), 222–259. 259 Table S3: Significance Levels for Correlations between Predictor Variables Variable Year Level Discipline Class Size Year Level - Discipline < 2.2e-16 - Class Size < 2.0e-16 < 6.24e-06 - Notes: Tested for correlations between the categorical variables (discipline and year level) using a chi‐square test and between each of the categorical variables and the continuous class size variable with an analysis of variance test. We used chisq.test and aov functions in the R programming language, respectively, for these tests. The table shows the adjusted p values for the respective test. Table S4: Multiple Linear Regression on WIL Scores Category Show & Tell Sell Engage Practice Out-of- classroom Coefficient ± SE (p-BH) Intercept 2.49 ± 0.42 1.65 ± 0.28 2.21 ± 0.37 2.37 ± 0.42 0.47 ± 1.76 Agricultural and Food Sciences 3.18 ± 0.55 (0.000)** 1.32 ± 0.36 (0.001)** 1.86 ± 0.49 (0.001)** 1.81 ± 0.55 (0.003)** 0.58 ± 1.92 (0.278) Biological and Marine Sciences 1.18 ± 0.65 (0.138) 0.62 ± 0.43 (0.229) 1.36 ± 0.58 (0.045)** 0.93 ± 0.66 (0.229) 0.1 ± 2.31 (0.841) Biomedical Science 1.37 ± 0.85 (0.185) 0.13 ± 0.56 (0.841) 0.93 ± 0.75 (0.278) 0.49 ± 0.85 (0.683) 0.03 ± 3.04 (0.948) Chemistry and Molecular Biosciences 0.97 ± 0.62 (0.185) 0.19 ± 0.41 (0.739) 0.87 ± 0.54 (0.185) 1.11 ± 0.62 (0.138) 0.18 ± 2.15 (0.739) Earth and Environmental Sciences 3.05 ± 0.55 (0.000)** 1.4 ± 0.36 (0.001)** 2.51 ± 0.49 (0.000)** 2.49 ± 0.56 (0.000)** 0.5 ± 1.93 (0.336) Veterinary Science 4.21 ± 0.71 (0.000)** 2.13 ± 0.47 (0.000)** 2.92 ± 0.63 (0.000)** 3.67 ± 0.71 (0.000)** 1.71 ± 1.95 (0.047)**