Literature Review Journal of Learning Development in Higher Education ISSN: 1759-667X Issue 11: April 2017 The big picture: mining the student data resource to inform learning development provision Elizabeth Tanguay Swansea University, UK Peter Hanratty Swansea University, UK Abstract Accurately assessing student needs is a key concern for learning developers in order for provision to have a strong evidence base. Studies investigating student study needs in higher education tend to provide a snapshot in time and are limited to a certain number of students (Liu et al., 2011; Sloan et al., 2013). Others have surveyed users of learning development centres, but not their needs (Buchanan, 2015). In order to develop evolving needs-based writing provision across Swansea University, the Academic Success Programme (ASP) has implemented a web-based system for recording identified areas for improvement in 1:1 tutorial sessions. The ongoing needs analysis is based on student essays in progress and identifies textual and functional categories for improvement. Grouping records according to student metadata allows us to profile student needs according to discipline area and year of study. The emerging patterns enable us to create bespoke workshops for particular student cohorts. To illustrate this process, an example needs analysis and curriculum design for Swansea University’s College of Human and Health Sciences is described. Keywords: learning development; study skills; needs analysis; writing provision; curriculum design; 1:1 appointments; tutorials; subject-specific support. Tanguay and Hanratty The big picture: mining the student data resource to inform learning development provision Journal of Learning Development in Higher Education, Issue 11: April 2017 2 Introduction While study support for non-native speakers (NNS) has a long history, support for native speakers (NS) has only recently gained importance in UK higher education, often under a study skills umbrella. There has been much written about the need for embedded or integrated subject specific provision of such skills courses (Wingate, 2015; McWilliams, 2014), with claims that generic ‘bolt on’ skills teaching is not as beneficial to student outcomes (McKendry et al., 2012). Where skills programmes are provided by a centralised department (integrated) rather than subject lecturers (embedded) (after McWilliams, 2014), a clear picture of the specific study needs of different student cohorts is essential. There have been comprehensive surveys of successful disciplinary student writing (for example, Nesi and Gardner, 2012). However, each institutional context is different, and feedback on current student work and where students’ abilities lie in relation to such exemplar texts, as well as which genres students struggle most with, can be more difficult to ascertain. The centrally-housed Academic Success Programme (ASP) at Swansea University, recently tasked with providing academic support to the entire student body, has implemented a system to gather information about student writing needs across the university on an ongoing basis. During 1:1 appointments, we have been using a form created through Google Sheets which allows us to collect and collate information. The data can then be analysed and used to inform our development of integrated courses for different colleges. This paper reports preliminary findings identified using a transferable needs analysis model for different cohorts of students and exemplifies bespoke writing workshop provision in the area of Human and Health Sciences. Context After a number of years providing in-sessional language support for international students, ASP was rebranded in 2013 with the extended remit of catering to students across all levels and disciplines. This extended remit meant that rather than providing primarily language-based support, there was now a need to understand the needs of an entire student body. For example, during the 2014/15 academic year our student profile comprised 6% international (European), 42% international (non-European) and 52% UK Tanguay and Hanratty The big picture: mining the student data resource to inform learning development provision Journal of Learning Development in Higher Education, Issue 11: April 2017 3 (of a total of 1595 students). Our support is divided into one-off generic workshops open to all students, longer courses and 1:1 appointments. A student will typically book a half-hour 1:1 appointment to discuss aspects of their study, most often bringing a sample piece of work with them. This piece of work is used by an ASP lecturer as a teaching aid. The student may have specific questions in mind with regard to their introduction, for instance. During the appointment, the ASP lecturer provides advice with regard to improvements which can be made. These interactions were historically recorded on paper and their collection primarily served the administrative task of recording student numbers for the centrally-funded centre. As such, records were seldom revisited. The introduction of a live, web-based record keeping system means we can collate and sort information, enabling us to use the data captured during our interactions with students as a robust, evolving evidence base for materials design. Already working closely with departments, we are developing courses based on aggregated feedback from our individual appointments in order to better meet our students’ needs by delivering bespoke courses. Needs analysis research in English for Academic Purposes Needs analysis aims at discovering what a student knows, what they don’t know, and what they need to know (Hyland, 2006). Gathering such information enables curriculum designers to assess the requirements of a particular set of students. However, needs analysis is rarely ongoing and more likely to provide a snapshot in time. Much needs analysis of writing skills in higher education has explored the needs of NNS (Liu et al., 2011; Deutch, 2003). There is a paucity of research on the needs of native speaker students although it is generally recognised that an ability to write successfully at university cannot be taken for granted (Wingate, 2012) and the pedagogical potential of English for Academic Purposes (EAP) for developing a mainstream model of instruction has been highlighted (Wingate and Tribble, 2011). Researchers have used a number of different methods to conduct needs analyses, perhaps reflecting the complexity of the task of understanding student needs. In examples of studies focusing on student or lecturer views on needs, researchers have used questionnaires to explore perceived needs and found differences in students’ needs perceptions and the courses they took (Liu et al., 2011). Among needs to be considered are the long-term global and the short-term personal goals (Deutch, 2003). Thus, as well Tanguay and Hanratty The big picture: mining the student data resource to inform learning development provision Journal of Learning Development in Higher Education, Issue 11: April 2017 4 as highlighting the fact that students are not necessarily best placed to pinpoint their own needs, the research highlights the fact that student needs change over time. Assessing student needs to inform curriculum design should be an ongoing process rather than one which provides a snapshot. Others have conducted corpus or text-based research in order to conduct present-to- target situation needs analysis (Hyland, 2006). There have been a number of useful insights gained as a result of research comparing novice and expert texts in off the shelf or bespoke academic corpora. Comparing student writing in The British Academic Written English Corpus (BAWE) with published academic texts, Chen and Baker (2010) found differences in the range of lexical bundles used by students as well as underuse of certain bundles. Similarly, Salazar (2014), comparing NS and NNS use of lexical bundles in her biomedical corpus, found that NNS relied on a limited number of high frequency bundles. Particular collocations that NNS students had problems with have been investigated and it would seem that neither length of stay in a country or number of years of study influences a student’s use of collocations (Nesselhauf, 2005). Flowerdew (2008) compared expert and non-expert problem solving texts in engineering to find certain differences in problem- solution patterning. However, while these textual analyses may be generalisable, they are static and based on a fixed data set rather than evolving to meet changing student global and personal needs. In other words, while such research has provided useful insights into the particular problems faced by particular students at particular times, we needed something that would be able to tell us about the ongoing and evolving needs of our ever- changing student cohorts. While the research outlined above highlights the complex nature of student needs analysis and provides information about the linguistic challenges that students face, it had the aim of helping NNS students. From the perspective of learning development (LD), the profile of students seeking help from learning development centres has been explored. Buchanan (2015) explored the profile of those who engaged with a LD service in terms of areas such as gender, module marks and parental education. However, the aim was to target underrepresented groups rather than to assess student needs. So, while LD centres are beginning to realise the potential benefits of profiling their users, they have not yet begun to explore student needs in the same way as researchers concerning NNS student writing. It seems that more work, especially into the writing needs Tanguay and Hanratty The big picture: mining the student data resource to inform learning development provision Journal of Learning Development in Higher Education, Issue 11: April 2017 5 of NS students across cohorts, is urgently required. This study, while also using a text (though not corpus) analysis, aims to assess the needs of all the students we see, irrespective of first language. This text-based analysis is an ongoing process conducted daily by lecturers as part of their work and, as such, the data is always current. The flexible tool being developed at Swansea University allows feedback from specialist study skills tutors to record both metadata about the students using the service and their particular needs. In collaboration with staff in academic departments, results are then translated into curriculum design. The ongoing nature of the project will allow needs analysis of an expansive group of students and has already resulted in a number of bespoke workshops for different student cohorts. Here, the study into the needs of the first year cohort of student nurses and the resulting workshop, as well as online material, is detailed. Methods In order to record and aggregate appointment data, a simple but secure online form has been authored using the free cloud-based platform, Google Sheets (see Figures 1.1 -1.3). The form automatically collates information into a spreadsheet which has enabled us to visualise how many students we see, their level of study and academic college. By sorting the dataset according to different identifiers, we are able to see trends in distinct student cohorts as to the common issues that their work presents and this has ultimately enabled us to tailor workshops to those specific student groups which provide solutions to the issues addressed. Each section of the form (for example, student number, level of study, college) is automatically assigned a column in a corresponding Google sheets spreadsheet and each entry on the form is assigned a new row under these columns. This transposition allows information concerning our 1:1 interactions with students to be collected, aggregated, updated and searched. The nature of this information spans from metadata concerning the interaction (such as date and time, student number, level of study, and degree programme) to textual features of the work presented. Tanguay and Hanratty The big picture: mining the student data resource to inform learning development provision Journal of Learning Development in Higher Education, Issue 11: April 2017 6 Figure 1.1. A screenshot showing the live form used in student appointments. Basic student Information The appointment form requires information from the students in the form of personally identifiable information, such as student number and name. Such information was traditionally collected as part of the department’s administrative record-keeping requirement (due to the centrally-funded nature of the service) and is not included in any analysis of the dataset. Information collected for needs analysis As well as the information above required for record-keeping, further categories were required in order to conduct a comprehensive needs analysis. These are the student’s college, level of study, genre of the work presented, and the nature of areas for improvement. Tanguay and Hanratty The big picture: mining the student data resource to inform learning development provision Journal of Learning Development in Higher Education, Issue 11: April 2017 7 Figure 1.2. Type of work. Figure 1.3. Areas to work on. The ‘type of work’ section (Figure 1.2) enables the lecturer to enter the genre of the text which the student presents, whereas the ‘areas to work on’ section is completed by the lecturer depending on the most frequent mistakes present. Both the ‘type of work’ drop- down list and the ‘areas to work on’ checkboxes are evolving and subject to refinement, depending largely on feedback from staff, as well as the contents of the ‘other’ option if selected. A comments section allows any other relevant comments to be made. In practice such comments are time-consuming to collate and have not been used in our analysis. Tanguay and Hanratty The big picture: mining the student data resource to inform learning development provision Journal of Learning Development in Higher Education, Issue 11: April 2017 8 Figure 1.4. Section of appointment data spreadsheet. Figure 1.4 shows a part of the system’s ‘back-end’. This is a cloud-based spreadsheet which is updated automatically each time the appointment form is completed. The data is collated and sorted according to college and level of study. In a growing dataset, the analysis of such qualitative data can be time consuming. By having an automatically filled spreadsheet, the administrative task of manually typing information into different fields is avoided and more staff time can be dedicated to using the collated information to help inform workshop design. Ethical considerations and data protection The data collected is anonymised in that student names and identifying numbers are not used in the data analysis. Permission is obtained from students and they can either opt in or out of having their anonymised data used for research purposes (see Figure 1.1). On a weekly basis, the data is downloaded to an offline master spreadsheet, deleted from the online platform and stored on a local hard drive. Tanguay and Hanratty The big picture: mining the student data resource to inform learning development provision Journal of Learning Development in Higher Education, Issue 11: April 2017 9 Results Once arranged by college and level of study, we are able to identify the real-time textual and linguistic trends presented during our appointments (Figure 1.2; Figure 1.3). Appointment data representing one of Swansea University’s colleges, the College of Human and Health Sciences (CHHS) is shown below: Figure 2.1. Appointments by level of study with students from the College of Human and Health Sciences at Swansea University. Figure 2.1 shows that in the College of Human and Health Sciences, first year undergraduate students have attended more appointments than students at other levels. Specifically, from 2014-2015 when there were 371 appointments with students from CHHS, almost half (49.5%) were with first year undergraduate students. This suggested that there was a real need for additional support for students at this level and that this would be a worthwhile focus of ASP’s limited resources. Tanguay and Hanratty The big picture: mining the student data resource to inform learning development provision Journal of Learning Development in Higher Education, Issue 11: April 2017 10 Figure 2.2. Percentage of text types seen according to level of study. Figure 2.2 shows that 37% of 371 interactions with students from the CHHS were with first year undergraduate students concerned about reflective writing. There were a smaller, but not insignificant, number of essays presented by first year students during appointments, with this accounting for 18% of interactions. This exploration of the text types, presented in conjunction with the level of study, shows that while students presented with a number of different text types, reflective writing for first year students is a key area for concern. Tanguay and Hanratty The big picture: mining the student data resource to inform learning development provision Journal of Learning Development in Higher Education, Issue 11: April 2017 11 Figure 2.3. Areas to work on’ results representing appointments with students from the College of Human and Health Sciences at Swansea University. As well as establishing which level of study and which genre to target, we also needed to focus provision on the linguistic elements of the genre with which students were having most difficulty, as shown in Figure 2.3. Here, it is important to note that the lecturer has the opportunity to select more than one area to work on during a 1:1 appointment. This means that a student may be advised on how to improve their essay structure and, for example, their use of grammar during a single interaction. Therefore, these figures show the number of times a problem was identified rather than numbers of students/ numbers of interactions. Particular attention to Figure 2.3 shows that of those first year undergraduate students that attended, the majority had issues with essay structure and using appropriate academic register (style/ formality) when writing. Implementation: findings to inform curriculum design When consulting with the CHHS about planning provision, this dataset was an invaluable resource for shaping our curriculum. Based on these findings, we developed a 2-hour workshop to be provided for the following intake of student nurses. The workshop Tanguay and Hanratty The big picture: mining the student data resource to inform learning development provision Journal of Learning Development in Higher Education, Issue 11: April 2017 12 presented how structure aids a reflective piece of writing, and provided examples of reflective frameworks that could be used to structure such pieces. Examples were gathered from module leaders within the CHHS so that materials used in the workshop were familiar to the students. As well as a focus on the structure of reflective writing, the workshop addressed the use of appropriate academic register within this genre. It seems that finding the right voice for a partially diaristic genre in an academic context can cause students difficulties. Specifically, students had trouble finding the appropriate academic register to express personal experiences, and develop these commonplace events to the abstracted analysis necessary to successfully complete the task. Twelve of these workshops were delivered during the 2015 Michaelmas term across two campuses, reaching an audience of 300+ students. Because these workshops were so well received, there has been higher demand for workshops than we have been able to cater for within the programme. An e-learning workshop based on the materials developed as a result of the needs analysis has since been developed in conjunction with the ASP e-learning officer and will be piloted with the next intake of students (see Appendix). This targeted e-learning workshop with videos and interactive exercises based on common areas for improvement will hopefully reduce demand on ASP’s limited resources. Limitations While the study reported here provided us with a number of useful insights into the particular needs of the cohort of first year CHHS students, a number of limitations with the current form became apparent. Firstly, the ‘grammar’ area to work on was too broad to be useful in providing targeted provision. This has since been changed to provide us with a more nuanced picture of student needs and show us in particular which areas of grammar are of the biggest concern (for example, subject-verb agreement/tense errors). The text types also proved to be limited and we continue to add additional categories as they become apparent. A further limitation with the form was that identifying the student’s college may be too broad to be able to adequately target provision. In order to combat this, we are looking to collect course of study information. Tanguay and Hanratty The big picture: mining the student data resource to inform learning development provision Journal of Learning Development in Higher Education, Issue 11: April 2017 13 The ‘self-selecting’ nature of the records is a further limitation. Only those students who seek help are recorded in this data. While a large proportion of the students presented with similar problems, it cannot be assumed that this dataset also represents those who did not attend, perhaps because they felt they did not need additional help, because they were unaware of the service, or because they were not able to book an appointment. However, the commonalities amongst students within the different student cohorts provide a useful starting point for designing bespoke workshops to be integrated in different courses, and the data here provides a robust basis with which to begin discussions with academics in the colleges. As the data set is updated on a daily basis, it is constantly growing (to date, we have a total of 1,281 interactions recorded). This means that the dataset can be consulted at any time to give a clear picture of current student needs. While the study here shows the analysis of a dataset from one college in the university, parallel analyses from the other colleges are now being conducted. Conclusion This research grew out of a real need to understand the needs of different student cohorts across Swansea University. The insights gained from the study into first year Human and Health Science students in particular provided us with a robust basis for curriculum design for the cohort of first year nursing students. The workshops on reflective writing implemented as a result of the study have been well received and are in high demand, leading to the creation of a parallel online resource for next year’s student intake. The means of understanding student writing needs analysis through mining a live tutorial dataset will now be replicated across different colleges of the university to gain a greater understanding of various student cohorts and their different needs. The model is also transferable across institutions which would allow for larger scale/comparative studies. Future directions include the continued evolution of the form with the further subdivision of categories, and an assessment of students’ perceived learning needs prior to the appointment, as well as recording essay titles. Tanguay and Hanratty The big picture: mining the student data resource to inform learning development provision Journal of Learning Development in Higher Education, Issue 11: April 2017 14 References Buchanan, A. (2015) ‘Anecdotal to actual: identifying users of learning development to inform future practice’, Journal of Learning Development in Higher Education, Issue 9, July, pp. 1-16 [Online]. Available at: http://www.aldinhe.ac.uk/ojs/index.php?journal=jldhe&page=article&op=view&path %5B%5D=295&path%5B%5D=pdf (Accessed: 29 June 2016). Chen, Y. H. & Baker, P., 2010. Lexical Bundles in L1 and L2 Academic Writing. Language Learning and Technology, 14(2), 30-49 Deutch, Y. (2003) ‘Needs analysis for academic legal English courses in Israel: a model of setting priorities’, Journal of English for Academic Purposes, 2(2), pp. 125-146. Flowerdew, L. (2008) Corpus-based analyses of the problem-solution pattern: a phraseological approach, Amsterdam: Philadelphia, John Benjamins Publishing Company. Hyland, K. (2006) English for academic purposes: an advanced resource book. London: New York, Routledge. Liu, J.-Y., Chang, Y.-J., Yang, F.-Y. and Sun, Y.-C. (2011) ‘Is what I need what I want? Reconceptualising college students’ needs in English courses for general and specific/academic purposes’, Journal of English for Academic Purposes, 10(4), pp. 271-280. McKendry, S., McKay, J. and O’Neill, D. (2012) ‘The role of an embedded learning development centre in enhancing the curriculum’, 9th Enhancement Themes National Conference. Heriot-Watt University, Edinburgh 7-8 March. McWilliams, R. and Allan, Q. (2014) ‘Embedding academic literacy skills: towards a best practice model’, Journal of University Teaching and Learning Practice, 11(3) [Online]. Available at: http://ro.uow.edu.au/jutlp/vol11/iss3/8 (Accessed: 29 June 2016). http://www.aldinhe.ac.uk/ojs/index.php?journal=jldhe&page=article&op=view&path%5B%5D=295&path%5B%5D=pdf http://www.aldinhe.ac.uk/ojs/index.php?journal=jldhe&page=article&op=view&path%5B%5D=295&path%5B%5D=pdf http://ro.uow.edu.au/jutlp/vol11/iss3/8 Tanguay and Hanratty The big picture: mining the student data resource to inform learning development provision Journal of Learning Development in Higher Education, Issue 11: April 2017 15 Nesi, H. and Gardner, S. (2012) Genres across the disciplines: student writing in higher education. Cambridge: Cambridge University Press. Nesselhauf, N. (2005) Collocations in a learner corpus. Amsterdam: John Benjamins Publishing Company. Salazar, D. (2014) Lexical bundles in native and non-native scientific writing: applying a corpus-based study to language teaching. Amsterdam: John Benjamins Publishing Company. Sloan, D., Porter, E. and Alexander, O. (2013) ‘Yes, you can teach an old dog new tricks. Contextualisation, embedding and mapping: the CEM model, a new way to define and engage staff and students in the delivery of an English language and study skills support programme: a case study of Heriot-Watt and Northumbria University’, Innovations in Education and Teaching International, 50(3), pp. 284-296. Wingate, U. (2012) ‘‘Argument!’ helping students understand what essay writing is about’, Journal of English for Academic Purposes, 11(2), pp. 145-154. Wingate, U. (2015) Academic literacy and student diversity: the case for inclusive practice. Bristol: Multilingual Matters. Wingate, U. and Tribble, C. (2011) ‘The best of both worlds? Towards an English for Academic Purposes/Academic Literacies writing pedagogy’, Studies in Higher Education, 37(4), pp.481-495. Author details Elizabeth Tanguay is Academic Success Programme Coordinator at Swansea University and a Senior Fellow of the Higher Education Academy. Peter Hanratty is Academic Success Programme Lecturer at Swansea University. Tanguay and Hanratty The big picture: mining the student data resource to inform learning development provision Journal of Learning Development in Higher Education, Issue 11: April 2017 16 Appendix Reflective writing workshop e-learning materials Materials developed by Frances Thornton Screenshot 1: The resource landing page. The accompanying audio guides students in using the resource as well as providing an introduction to the specific reflective piece students are required to write. It also notes that the word counts here are approximations only. Tanguay and Hanratty The big picture: mining the student data resource to inform learning development provision Journal of Learning Development in Higher Education, Issue 11: April 2017 17 Screenshot 2: A part of our electronic resource highlighting appropriate pronoun use in different sections of reflective writing. Screenshot 3: Academic register. The accompanying audio, using relevant examples, discusses how students can employ an appropriate register in their reflective writing. Tanguay and Hanratty The big picture: mining the student data resource to inform learning development provision Journal of Learning Development in Higher Education, Issue 11: April 2017 18 Screenshot 4: Academic register continued. Screenshot 5: Academic register continued. Tanguay and Hanratty The big picture: mining the student data resource to inform learning development provision Journal of Learning Development in Higher Education, Issue 11: April 2017 19 Screenshot 6: Examples of appropriate evaluative language for use in reflective writing.