D Raftery & A Risquez 24 Engaging students through the VLE: comparing like with like using the #VLEIreland student survey Damien Raftery, Institute of Technology Carlow Angelica Risquez, University of Limerick1 Abstract Virtual learning environments (VLEs) provide the core infrastructure for the digital learning experiences for many students in Irish and international higher education institutions. Hence the student experience and voice offer an important perspective to understand to what extent this proves a strategic choice, and investment of institutional resources and lecturers’ time. However, surveys comparing a diversity of institutions at different moments in time offer serious methodological limitations to data analysis. In this paper, results of the #VLEIreland student survey are presented for four Irish higher education institutions, which were surveyed using the common questionnaire with 3,332 student responses in 2011 and 5,170 when the survey was repeated in 2013. To some extent, this allows us to analyse the data while relatively controlling for institutional influences. This snapshot of students’ use and perceptions at two points allow for tentative trends to be drawn as to any changing patterns over time. The comparison of results indicate a consistent finding of high frequency of use (with almost half using daily in the more recent findings). The VLE thus provides a stable base for sharing learning resources, managing assignments and student communications. The consistency of the findings indicate a certain maturity of use of the VLE, albeit with a continued emphasis on disseminating notes and other learning resources to students. As the student VLE experience is largely determined by their lecturers’ use, a continued emphasis on the development of digital capacity of lecturers (as well as students) is needed. Irish Journal of Technology Enhanced Learning Ireland, 2018. © 2018 Raftery, D. & Risquez, A. The Irish Journal of Technology Enhanced Learning Ireland is the journal of the Irish Learning Technology Association, an Irish-based professional and scholarly society and membership organisation. (CRO# 520231) http://www.ilta.ie/. This is an Open Access article distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license. 1 The basis for this article stems from a conference paper (Cosgrave et al., 2013) and we would like to acknowledge the valuable contributions of all our collaborators. Irish Journal of Technology Enhanced Learning Vol 3, Issue 2, 2018 25 1. Introduction In Ireland, the use of digital tools and the internet both within and beyond formal teaching environments has become a normalised and expected dimension of the teaching and learning experience (O’Rourke, 2017, p.13). In Irish higher education institutions as well as internationally, virtual learning environments (VLEs) have matured from early pilots in the early 2000s to now being widely used and considered core infrastructure. To justify this widespread investment in VLE technology, it is important to study patterns of actual student VLE use and student satisfaction (Naveh, Tubin & Pliskin, 2010) and indeed it was commented that Irish higher education lacks an evidence base in relation to VLE usage (Devine, 2015, p.16). It is essential to allow the student voice to inform the use of VLEs and the continuous improvement of their learning experiences. Irish students, along with staff, have an expectation of “ubiquitous connectivity, access to online resources and the availability and use of digital tools in almost every aspect of their work” (O’Rourke, 2017, p.21). This paper contributes to this gap in the literature. Following a summary of research methods, this article presents and discusses students’ views on the use of VLEs across higher education in Ireland. In particular, we focus our analysis on a comparison between the responses from students of four Irish higher education institutions which took the survey simultaneously in a two year gap (in 2011 and 2013 respectively). To some extent, this allows us to analyse the data while relatively controlling for institutional influences. These snapshots of students’ use and perceptions at two points allow for tentative trends to be drawn as to any changing patterns over time. Prior to conclusions, the comparison analysis is discussed along with other research and some implications are drawn. 2. Methods As described elsewhere in this special issue (Ryan & Risquez, 2018), students were surveyed using the standard questionnaire with a common set of questions. To enable comparisons over time, changes have been kept to a minimum in the survey through the multiple administrations year on year. In this article, the analysis compares the 3,332 student responses in 2011 for four institutions in which the survey was repeated in 2013, at which time 5,170 students responded. The four colleges represent both universities and institutes of technology, and a variety of VLEs are used. The response rates in the four colleges varied and, as the survey was administered online, the sample is self-selecting leading to potential sample bias. A specific limitation in interpreting any change between 2011 and 2013 overall for the four colleges is that there has been no attempt to control for changes in the relative composition of the overall sample (proportion of respondents from each institution in 2011: 38%, 31%, 18% and 13%; in 2013: 37%, 39%, 19% and 6%). Similarly, there may be other changes in demographics or the nature of the students surveyed that may distort the overall change reported (for example, gender and age of respondents from each institution in 2011: 42% female, 63% aged 23 and under; in 2013: 56% female, 69% aged 23 and under). In part, due to differences in VLE terminology, there may be substantial non-responses to certain questions. These limitations are a caveat to the interpretation and generalisability of the findings presented. This being said, the overall sample surmounts 8,400 responses, which offers a degree of reliability to the data presented next. D Raftery & A Risquez 26 3. Findings The 2013 student responses are, on average, strikingly similar to those of 2011 indicating relatively stability in student attitudes and use of the VLE. There seems to be a general increase in the routine usage of the VLE, with the continued predominant, almost universal, use of the VLE to solve the problem of disseminating learning resources to students. The main difference is in where and when students are accessing the VLE from, with a marked increase in the use of mobile devices. There is also a continued increase in the use of the VLE for submitting assignments and for academic integrity/plagiarism detection. The analysis below is presented under key themes that emerged in previous reporting of our research including the VLE as a content distribution platform, the need for consistency of use, the potential to enhance student communication and the expansion of flexibility (Risquez et al., 2013) and can be compared with results reported in the paper by Ryan and Risquez (2018) in this Special Issue. 3.1 VLE as a content distribution platform If I can't attend college because of being sick, or family commitments, I don't feel completely lost as I can keep track of everything in [the VLE]. Makes me revise more and allows me to prepare for labs more efficiently as my lecturer posts videos online in [the VLE]. Students use the VLE frequently. This is consistent: slightly up in 2013, 48% of respondents use the VLE daily with a further 40% using it a few times a week and 7% once a week. This compares to 43% using daily in 2011, with 42% a few times a week and 7% once a week. Reported use % Broadly useful % 2011 2013 2011 2013 Lecture notes/handouts 89 99 87 96 Other course documents 94 97 87 90 Reading lists & online learning resources 68 90 60 77 Submitting assignments 81 89 76 83 Plagiarism detection/Turnitin 58 70 46 54 Class announcements 75 85 67 65 Online discussions 47 56 30 28 Quizzes 56 62 44 45 Table 1: Proportion of students self-reported use and rating as broadly useful (useful or very useful) The basic use of the VLE is as a simple, stable transmission system for lecture notes and other learning resources. As can be seen in Table 1, there is a general increase in reported VLE use across all categories asked. A substantial majority of students reported that they found the VLE broadly useful for getting course material from the lecturer – either lecture Irish Journal of Technology Enhanced Learning Vol 3, Issue 2, 2018 27 notes/handouts (96% in 2013, up from 87% in 2011) or other course resources (90% in 2013, up from 87% in 2011). There is substantial and continued growing use of the proportion of students who report finding the VLE useful for submitting assignments (83% in 2013, up from 76% in 2011) and for academic integrity/plagiarism detection (54% in 2013, up from 46% in 2011). There is stable rating of usefulness (about 2 out of 3) of class announcements. Respondents consistently had high levels of agreement with the statement that the VLE gives them more access to resources and learning materials (89% in 2013, 88% in 2011). The VLE provides a content management system with increasing use for an efficient process for managing and grading assignments and as an additional communication forum for students. The increased submission of assignments may be due to the integration of systems such as Turnitin within the VLE. Despite being substantially more demanding of lecturer time, there seems to be a substantial minority finding quizzes and online discussions useful, a finding consistent with the findings of surveys with lecturers reported in this issue (Farrelly, Raftery & Harding, 2018). 3.2 Consistency of use Only one of my lecturers uses [the VLE] which is great for that subject. Unfortunately I don't get to use it for other subjects. Some lecturers put material up in a fairly disorganised manner which can be frustrating when trying to access material quickly. Students consistently agree that the VLE is helpful and easy to use, about 94% and 83% respectively in both 2013 and 2011. There was a general increase in the proportion of coursework that had material in the VLE from 2011 to 2013. 22% of respondents in 2013 reported all their coursework had material in the VLE compared to 10% in 2011 whilst there was a reduction having none to 2% in 2013 from 7% in 2011. Only a small proportion of respondents reported not using the VLE, about 1% in both 2013 and 2011. The overwhelming barrier given as a reason is the lack of use of the VLE by lecturers rather than any technical issues such as system reliability or access. Students clearly would like their lecturers to make more use of the VLE: in 2013, 68% of respondents still strongly agreed or agreed with this statement (consistent with the 69% in 2011). Students want comprehensive and consistent basic usage that facilitates easier navigation within the VLE. This has implications for academic development, to ensure all staff are consistently using the basics. There is a modest increase in the level of agreement that lecturers make good use of the VLE, from 57% in 2011 to 62% in 2013. This continues an upward trend, implying that digital capacity is gradually being developed in the academic community and this needs to continue. Indeed, when asked “What is the most important thing you would like to see improved about the VLE?” in 2013 56% of students chose better usage by lecturers compared to 68% in 2011. Note that the second most popular choice is consistently more use of audio/multimedia, which rose to 19% in 2013 from 13% in 2011. There was also a small increase for making the VLE easier to use, from 8% in 2011 to 13% in 2013. Coupled with a consistent 6% level of disagreement that VLEs are easy to use and about 10% disagreement that the respondent can get adequate help and support to use the VLE, this indicates that supporting students in using the VLE remains important. D Raftery & A Risquez 28 3.3 Enhancing and sustaining student communication My course is very time consuming and it's difficult to manage study around this. [The VLE] allows me that time to concentrate my study in a precise way. It suits me just the way it is ... It provides you with a good picture of how the semester will look regarding assignments, thus allowing adequate planning. VLEs facilitate students communicate and engage with their lecturers and each other. There is a consistent agreement that the VLE gives more access to lecturers (70% in 2011, 67% in 2013). Relatively few students feel that using the VLE makes it more likely for them to communicate with their lecturer, for example by asking a question online (25% in 2011, 22% in 2013). However the VLE may be a valuable forum for students unwilling or unable to communicate with lecturers in class or in person, with the VLE offering an ‘open office door’ for students to engage with their lecturers. Note that consistently a much smaller proportion of respondents felt the VLE increased their access to classmates (27% in 2011, 21% in 2013) but again this may be valuable to particular students. Figure 1 Aspects of the VLEs perceived contribution to learning As can be seen in Figure 1, students seem to feel that the VLE provides a genuine aid for learning. There is a consistent majority that report that using the VLE makes it easier for them to learn (67% in 2011, 72% in 2013), as well as adding to and clarifying what is covered in class. In addition, almost half feels the VLE helps them understand how well they are doing (44% in 2011, 41% in 2013). Irish Journal of Technology Enhanced Learning Vol 3, Issue 2, 2018 29 3.4 Meeting flexibility requirements I have access to [the VLE] from my phone. All notes are portable and when I have a free moment I can go online and study. I can learn at home, even in bed. Figure 2: Where students access the VLE from Most respondents access the VLE from home as can be seen in Figure 2, with substantial numbers accessing from laptops on campus and from open-access PCs. The biggest change is in the use of mobile devices to use the VLE: in 2013 41% accessed from a mobile phone and 14% from a tablet device, compared to 14% from mobile device in 2011. The rapid change in technology has major implications for the provision of wi-fi on campus and the use of such devices to support learning both inside and outside of the classroom. In 2013, 36% reported using wireless access outside campus compared to 5% using an internet café in 2011. (Note that in 2013 survey the options related to internet café and mobile devices were updated to reflect changes in technology use.) Respondents were asked to indicate what times they used the VLE. Although such self- reporting is much less reliable than access logs, there was a dramatic increase from 2011 to 2013 in the level of access reported across a broad range of hours. As well as high usage during the day, this change is most pronounced from 7am to 9am (from 7% in 2011 to 21% in 2013), from 6pm to 10pm (from 36% to 81%) and during the night from 10pm to 7am (from 10% to 29%). This increase is largely outside of conventional study and office hours and may have implications for support. This increased flexibility of location and time (anywhere, anytime), coupled with broadly positive student attitudes that the VLE is helpful and makes learning easier, is a key outcome of the use of VLEs. Although beneficial for all students the D Raftery & A Risquez 30 remote availability of VLEs is particularly important for part-time and distance learners, enabling ubiquitous access to learning materials and communication tools. Mobile access, including VLE apps, provide useful new ways to easily communicate with students and enable the use of the VLE to support learning activities in class (see Raftery (2018) in this issue for further discussion). 4. Discussion [The VLE] has been very important to my studies throughout the years, when on Erasmus the lack of a useful and easily accessible system like [the VLE] was very noticeable. I feel that the use of online quizzes and video links on material covered in lectures helps with my learning of subjects and feel that all lecturers should be making more use of this feature. These quotes illustrate aspects of how students value the VLE and its positive impact on their learning. The comparative analysis in the findings have highlighted the stability in student attitudes and use of the VLE. There has however been a general increase in the routine usage of the VLE, with a substantial increase in the student use of mobile devices. The VLE’s predominant role as a content distribution platform has been complemented with increasing use of the VLE for managing and grading assignments. The comparative analysis above confirms a stability in students’ view on the need for lecturers to use the VLE consistently, and highlights how the VLE enhances student communication and flexibility. Survey respondents consistently consider that the VLE is helpful (about 94%), consistent with findings of Heaton-Shrestha et al. (2009) that, in a UK university, students felt the VLE enhanced their performance, effectiveness and experience. Individual students commented that “[the VLE] gives you direction”, “I’d be lost without [the VLE]. It’s really helpful” and “I could not survive academically without [the VLE]”. There is stable agreement that the VLE is easy to use (over 80%), but small levels of disagreement indicate continued importance of appropriate initial induction and ongoing support. Students like the flexible and structured access to learning resources that the VLE offers – being able to access from wherever and whenever, with substantial access from home and in the evenings. Consistency in the layout, terminology and design of courses within the VLE is wanted and valued by students. The increased use of mobile devices has implications for campus infrastructure (such as the provision of wi-fi and flexible learning spaces) and VLE design (including apps), as well as demand for mobile access via the VLE to rich multimedia learning resources (Irish examples: Marcus-Quinn & Cleary, 2015; Mitchell, 2014; Fannon, Kelly, MacBlain, Raftery & Brennan, 2015). There is a general increase in students’ reported VLE use. A key determinant of students’ use of the VLE is the amount and type of use by lecturers. An increasing majority of respondents agreed that their lecturers made good use of the VLE, with consistently about 7 out of 10 wanting their lecturers to make more use of the VLE. Despite heavy workloads (especially in context of increased teaching and administration including working with more students), academics are demonstrating increasing effectiveness in basic VLE use. This can be understood as part of a trend where lecturers tend to make, on average, only incremental changes to their practice when faced with new technology in the form of a VLE (Dutton, Cheong & Park, 2004; Jenkins, Browne & Walker, 2005; Kirkup & Kirkwood, 2005). Naveh et al. (2010) suggest that “instructors can maintain their conservative teaching habits except Irish Journal of Technology Enhanced Learning Vol 3, Issue 2, 2018 31 for posting their course content … this can be done at low cost, yielding relatively high student satisfaction” (p. 132). This is consistent with Brown’s argument that, despite almost universal adoption by institutions in the UK, “VLEs have had only a relatively slight impact on pedagogy in higher education” (2010, p.7). For continual improvements as well as substantial or innovative changes in teaching practice, both national and institutional strategic vision is needed, coupled with a renewed emphasis on fostering digital literacies, academic professional development, and pedagogical and technical support (for further discussion of staff attitudes to, and use of, the VLE see Farrelly, Raftery and Harding (2018) and of the importance of staff development see Harding (2018), both in this Special Issue). Indeed, the National Forum report Building Digital Capacity in Higher Education directly links the under-utilisation of current digital infrastructure with the need for strategic and pedagogical guidance, and the development of digital literacy within the teaching community as follows (2014, p.7): The constant emergence of newer and better tools has often resulted in confusion among staff regarding the best tools to recommend and use, and to develop and learn about in their teaching and learning practices. This may go some way towards explaining the emerging evidence suggesting that key digital resources (for example, virtual learning environments) are not being used to their full pedagogical capacity. Our findings that in 2013 almost 9 in 10 students report finding the VLE useful for submitting assignments has been corroborated by findings in the Irish Survey of Student Engagement, where 86% of respondents stated they used a VLE to discuss or complete an assignment sometimes, often or very often (2015, p. 18). Thus as the VLE is increasingly used to manage aspects of assessing students (submission of assignments, online quizzes, discussions and journals), the administrative requirements to retain student work mean that formalised policies and procedures are required. Policies are also needed to ensure adequate privacy and data protection, whilst enabling analysis of student data within the VLE (such as a data source for learning analytics projects such as DESSI (National Forum, n.d.); see Logan-Phelan (2018) in this Special Issue for further discussion of learning analytics). 5. Conclusion The consistent finding of high frequency of use (with almost half using daily and a further four in ten students using the VLE at least a few times a week in 2013), and the greater flexibility of access and communication via mobile devices, combines to reinforce that the VLE provides a stable base for sharing learning resources, managing assignments and student communications. The consistency of the findings indicates a certain maturity of use of the VLE. As indicated by the rapid rise in the use of mobile devices to use the VLE, advances in technology and the uses of that technology can facilitate major and profound changes in practice and use. However, this is by no means guaranteed. Indeed technological developments, combined with wider changes in higher education, will affect the evolving nature of use and future role of VLEs, perhaps disruptively. In this context, national and institutional strategic engagement is important as is that of departments, and individual and groups of academics. The support and continued academic professional development of D Raftery & A Risquez 32 lecturers is needed with appropriate emphasis on digital literacy and digital pedagogy to enhance student learning. Irish Journal of Technology Enhanced Learning Vol 3, Issue 2, 2018 33 References Brown, S. (2010). From VLEs to learning webs: the implications of Web 2.0 for learning and teaching. Interactive Learning Environments, 18(1), 1-10. Cosgrave, R., Risquez, A., Raftery, D., Logan‐Phelan, T., Costello, E., Harding, N., Palmer, M., McAvinia, C. & Farrelly, T. (2013). The VLE Usage Survey Five Years in: Overview of Findings and future directions. EdTech 2013, research paper presentation, University College Cork, 30-31 May. Devine, J. (2015). Strategic and Leadership Perspectives on Digital Capacity in Irish Higher Education. Dublin: National Forum for the Enhancement of Teaching and Learning. Retrieved from https://www.teachingandlearning.ie/wp-content/uploads/2014/03/Senior- Management-FINAL.pdf Dutton, W. H., Cheong, P. H. & Park, A. (2004). An ecology of constraints on e-learning in higher education: the case of a virtual learning environment. Prometheus, 22, 2, 131–149. Fannon, F., Kelly, O., MacBlain, M., Raftery, D. & Brennan, S. (2015). Cross-institutional development of sharable learning technology tools for flexible accounting education. EADTU 2015 Conference Proceedings. Farrelly, T., Raftery, D. & Harding, N. (2018). Exploring lecturer engagement with the VLE: findings from a multi-college staff survey. [Special Issue, McAvinia & Risquez]. Irish Journal of Technology Enhanced Learning, 3(2), 11-23. Harding, N. (2018): The Digital Turn: staff perceptions of the virtual learning environment and the implications for educational developers. [Special Issue, McAvinia and Risquez]. Irish Journal of Technology Enhanced Learning, 3(2), 58-76 Heaton-Shrestha, C., May, S., & Burke, L. (2009). Student retention in higher education: what role for virtual learning environments? Journal of Further and Higher Education, 33(1), 83-92. Irish Survey of Student Engagement (2015). Retrieved February 2nd 2018 from http://studentsurvey.ie/wp-content/uploads/2015/11/ISSE-Report_2015-final-tagged.pdf Jenkins, M., Browne, T. & Walker, R. (2005). VLE Surveys: A longitudinal perspective between March 2001, March 2003 and March 2005 for higher education in the United Kingdom (Vol. 30). Oxford, England: Universities and Colleges Information Systems Association (UK). Kirkup, G. & Kirkwood, A. (2005). Information and communications technologies (ICT) in higher education teaching—a tale of gradualism rather than revolution. Learning, Media and Technology, 30, 2, 185–199. Logan-Phelan, T. (2018). The Buzz around Learning Analytics – enablers and challenges identified through the #VLEIreland Project. [Special Issue, McAvinia & Risquez]. Irish Journal of Technology Enhanced Learning, 3(2), 77-85. Marcus-Quinn, A. & Y. Cleary (2015). How Irish Postgraduate Students Use Mobile Devices to Access Learning Resources. In Y. Zhang, Handbook of Mobile Teaching and Learning. Berlin: Springer Berlin Heidelberg, pp. 1-14. https://www.teachingandlearning.ie/wp-content/uploads/2014/03/Senior-Management-FINAL.pdf https://www.teachingandlearning.ie/wp-content/uploads/2014/03/Senior-Management-FINAL.pdf http://studentsurvey.ie/wp-content/uploads/2015/11/ISSE-Report_2015-final-tagged.pdf D Raftery & A Risquez 34 Mitchell, C.A. (2014). Using VLEs and mobile technology to aid in the provision of an upgraded delivery mechanism for the basic principles of quantity surveying, 6th International Conference on Education and New Learning Technologies, EDULEARN14, Barcelona (SPAIN), 7th, 8th and 9th July, 2014. National Forum (2014). Building Digital Capacity in Higher Education. Principles and First Insights from the Sectoral Consultation on Building Digital Capacity in Irish Higher Education.: National Forum for the Enhancement of Teaching and Learning. Retrieved from http://www.teachingandlearning.ie/wp-content/uploads/2014/05/Digital-Roadmap- PHASE1MAY282014.pdf. National Forum (n.d.). Data-Enabled Student Success Initiative (DESSI). National Forum for the Enhancement of Teaching and Learning. Retrieved from https://www.teachingandlearning.ie/learning-analytics-educational-da ta-mining-learning- impact/dessi-information-booklet/ Naveh, G., Tubin, D. & Pliskin, N. (2010). Student LMS use and satisfaction in academic institutions: The organizational perspective. The Internet and Higher Education, 13(3), 127- 133. O’Rourke, K. C. (2017) Ireland’s Higher Education Technical Infrastructure: A review of current context, with implications for teaching, and learning enhancement. National Forum Report. Retrieved from https://www.teachingandlearning.ie/wp- content/uploads/2017/12/Final-Infrastructure-report-with-doi-web-ready.pdf. Risquez, A., McAvinia, C., Raftery, D., O’Riordan, F., Harding, N., Cosgrave, R., Logan- Phelan, T. & Farrelly, T. (2013). An investigation of students' experiences of using virtual learning environments: implications for academic professional development. In C. O'Farrell & A. Farrell (Eds.). Emerging Issues in Higher Education III: from capacity building to sustainability. Athlone: EDIN (Educational Developers in Ireland Network). Retrieved from http://www.edin.ie/?page_id=112 Raftery, D. (2018). The VLE and mobile learning. [Special Issue, McAvinia & Risquez]. Irish Journal of Technology Enhanced Learning, 3(2), 66-76. Ryan, D. & Risquez, A. (2018). ‘Lessons Learnt’: the student view in the #VLEIreland project [Special Issue, McAvinia and Risquez]. Irish Journal of Technology Enhanced Learning, Vol 3(2), 1-10. http://www.teachingandlearning.ie/wp-content/uploads/2014/05/Digital-Roadmap-PHASE1MAY282014.pdf http://www.teachingandlearning.ie/wp-content/uploads/2014/05/Digital-Roadmap-PHASE1MAY282014.pdf https://www.teachingandlearning.ie/learning-analytics-educational-da%20ta-mining-learning-impact/dessi-information-booklet/ https://www.teachingandlearning.ie/learning-analytics-educational-da%20ta-mining-learning-impact/dessi-information-booklet/ https://www.teachingandlearning.ie/wp-content/uploads/2017/12/Final-Infrastructure-report-with-doi-web-ready.pdf https://www.teachingandlearning.ie/wp-content/uploads/2017/12/Final-Infrastructure-report-with-doi-web-ready.pdf http://www.edin.ie/?page_id=112