Microsoft Word - [1]629-4233-3-CE-final[1][1].doc Australasian Journal of Educational Technology, 2014, 30(1). ascilite 1 Using research to inform learning technology practice and policy: a qualitative analysis of student perspectives Carol Russell University of Western Sydney, Australia Janne Malfroy Australian Catholic University, Australia Maree Gosper Macquarie University, Australia Jo McKenzie University of Technology Sydney, Australia As learning technologies are now integral to most higher education student learning experiences, universities need to make strategic choices about what technologies to adopt and how to best support and develop the use of these technologies, particularly in a climate of limited resources. Information from students is therefore a valuable contribution when determining institutional goals, building infrastructure and improving the quality of student learning. This paper draws on a survey of student experiences and expectations of technology across three Australian universities. Analysis of text responses from 7,000 students provides insight into ways that institutional learning technologies and academic- led technologies are influencing the student experience. This paper also discusses how the three universities have used this information to develop strategic initiatives, and identifies a need for new strategies to support academic-led use of the available tools. Introduction Digital technologies in higher education The role of digital technologies was highlighted in a recent report on Australian university business models, which cited “digital technologies” as one of the five key challenges facing universities, claiming that “Digital technologies will transform the way education is delivered, supported and accessed, and the way value is created in higher education and related industries” (Ernst & Young, 2012, p. 9). The report forecasts the need for universities to adopt a more strategic and innovative approach to the use of digital technologies in the future. Australian universities invest heavily in learning technologies, which have had quite considerable take-up by both on-campus and off-campus students. However, often the growth has been less due to well-planned curriculum approaches with organised teamwork, and more dependent on an approach based on individual teachers’ interests and capabilities (Forsyth, Pizzica, Laxton, & Mahony, 2010). Developing strategic plans for the integration of learning technologies is complex in a higher education environment where there is competition for resources. Empirical evidence is important to counterbalance the wealth of claims about educational technology, many of which have failed to be implemented effectively and have therefore failed to deliver (EdTech Now, 2012). There is also an imperative in this decision-making process to understand the academic and social practices and preferences of students (Kennedy, Judd, Churchward, Gray & Krause, 2008). In another Ernst and Young (2011) report on “Higher Education and the power of choice”, the authors highlight the impact of recent changes to the funding model in Australian universities and argue that: The introduction of a demand-driven funding model and other forms of deregulation will shift power to the consumer. Universities will need to build institutions that clearly understand which consumers they are targeting, what those consumers need, and how they prefer to be serviced. (Ernst and Young 2011, p. 8) Australasian Journal of Educational Technology, 2014, 30(1). 2 Whilst many of us are reluctant to call our students “consumers”, there is no doubt that their experiences and expectations about the success of ICT interventions for learning will be key to student success and retention at university (JISC, 2011). A study of student engagement in the UK concluded that “… more emphasis seems to be placed on viewing students as consumers and rather less on viewing students as partners in a learning community”, and that there is scope for involving students better as “partners in a learning community” (Little, Locke, Scesa, & Williams , 2009). It was the desire to include the Australian university student voice in the planning and development of educational technology services that led to a survey in 2010, conducted across three large metropolitan Australian universities. The survey featured both quantitative and qualitative questions, and provided empirical evidence on students’ experiences and expectations for use of technology to support their learning. The quantitative results have been published elsewhere (Gosper, Malfroy, & McKenzie, 2013). This paper examines the qualitative results from the survey, and discusses how the three universities have used the survey information to develop strategic responses. Cross-insitutional Survey The Student Experience and Expectation of Technology (SEET) survey was a collaboration involving three large Sydney-based universities drawing on slightly different demographics and student cohorts. The SEET survey was conducted in 2010 and the combined responses from all three universities yielded a total data set of 10,269 participants: 7,419; 1,754 and 1,104 from each institution respectively. Demographic information provided by 75% of these respondents showed that the sample was reasonably consistent with the university populations, in terms of: • gender (43.8% female, 31.1% male); • age (54.3% 25 years or less, 13.1% between 26 and 50 years, and 1.5% over 50); • study mode (63.6% full-time, 11.3 % part-time); • study level (61.6% undergraduate and 13.2% postgraduate). The survey design was informed by other major student surveys used in the UK (Ipsos MORI, 2008), the US (Educause, 2009 & 2010) and Australia (Kennedy et al., 2008). The aim of the SEET survey was to identify: • technologies currently used for work and social purposes; • technologies currently used for study purposes; • technologies students would like to use more in the future; • satisfaction with services and facilities; • the extent to which experiences and expectations vary across different cohorts of students defined, for example, by discipline grouping, age, gender, level of study, enrolment mode and equity status. The survey was open to all students from first year to PhD level and included 137 quantitative (multiple choice) questions and four open-ended questions to elicit qualitative data. The aim was to identify context-specific patterns across three Australian universities, which we were then able to compare with broader generational changes taking place among university students worldwide. This mixed-method approach fits with the pragmatic nature of an inquiry designed to inform university planning (Creswell, 2009, pp. 17-18). Analysis of the quantitative data from the survey suggested a multidimensional approach to planning comprised of three different levels of technology provision (Gosper et al., 2013): i. institutionally supported technology provision; core systems, such as the institutional online learning management systems and campus IT infrastructure and helpdesk services; ii. academic-led technology provision; technologies that are chosen or created by teachers to support specific learning activities in their disciplines, such as online quizzes, interactive simulations or online discussion forums; iii. student-led technology provision; technologies students choose to source and use to support their learning, such as Google Docs, Facebook groups and their own mobile devices. Australasian Journal of Educational Technology, 2014, 30(1). 3 The four open-ended questions allowed students to describe in their own words what they valued and what they wanted their universities to provide. About 70% of the respondents provided substantive text comments, yielding 13,500 student comments. A preliminary qualitative analysis of these text comments for each university gave further insight into how students felt technology enhanced their learning. The terms “accessibility” and “flexibility” were used frequently when discussing the benefits of online learning and we hoped that the text responses would show more explicitly what students meant when they used these terms. We therefore carried out a more systematic qualitative analysis to gain insight into what students specifically like about these learning technologies, how they contribute to the learning experience and how their use might enhance learning. This is reported in the first half of the paper and in the second half, we report on how, in the two years since the data was gathered, the survey results have contributed to planning, infrastructure and professional development initiatives. Methodology and research methods Our overall methodological framework broadly follows the critical realist paradigm as described by Ackroyd (2009), focusing on the interaction of people with organisational and technical systems. This a pragmatic methodology that involves a questioning and reflexive approach to theory (Alvesson & Sköldberg, 2000). We are not relying solely on statistical evidence, as in large-scale international studies like the annual Educause survey (Dahlstrom, 2012) which uses a quantitative instrument to track broad trends over time. Nor are we focusing only on qualitative analysis to explain contextual social experiences in depth. Instead we are following a convergent mixed methods research design (Creswell & Plano-Clark, 2011, p. 180) in which quantitative and qualitative data are collected in parallel from the same sources. This paper focuses on analysing the qualitative survey data to review how the universities’ strategies and policies are addressing student concerns. Table 1 shows the total number of substantive responses (i.e. at least one relevant word other than “none” or ‘n/a’) to each of the four open-ended question across all three universities. The qualitative analysis adds depth and enables us to check whether the quantitative findings were missing some underlying linkages or systemic influences. Whereas the quantitative analysis suggests three levels of planning for learning technology provision (institution-led, academic-led and student-led), here we are exploring how students may be linking these and other aspects of their experiences and expectations of technology at university. Table 1 Qualitative data available for analysis Question Number of responses A Please describe the most important ways that technology has assisted your learning at University 6,876 B Please describe ways in which the University could use technology to better support your learning 6,610 C What has technology enabled you to do that you wouldn't otherwise be able to do at University? 6,162 D Please add any other comments you would like to make about your experiences and expectations of technologies at University 4,192 Preliminary analyses of samples from each university’s student responses to these questions were carried out by researchers from each of the universities. The three separate analyses showed similar patterns between universities and also found that responses to questions C and D repeated themes found in responses to questions A and B. We therefore pooled the data from the three institutions and carried out a more detailed content analysis of all the responses to questions A and B. The processes used were iterative, with decisions at each stage reviewed with co-authors and made explicit, as outlined below. Reflexivity requires acknowledging that student comments will have been in part directed by the topics raised in the quantitative questions. Nevertheless, the analysis of the text comments used a more grounded approach that allowed for identification of unanticipated themes and patterns. Australasian Journal of Educational Technology, 2014, 30(1). 4 The content analysis methods we used are described by Franzosi (2004); in particular, word frequency analysis and thematic analysis. The results provide evidence for modelling some of the underlying processes shaping student experience of technology in university learning. A number of different analytical features of NVIVO software were used in sequence to analyse text responses. As an initial pilot – and also to guide institutional strategies – each institution’s data was analysed separately. This initial analysis involved running word frequency queries on the responses to each question followed by manual coding (by different researchers for each university’s data) of a representative sample of the text for themes. There was little difference in themes identified between coders and institutions, either in the word frequency results or in the themes identified. The analysis reported here, summarised in Figure 1, is the subsequent analysis of the combined data from all three institutions. While one researcher took the lead in carrying out the NVIVO analysis, the resulting themes and relations between these examples and the themes were confirmed by researchers from the three institutions. The stages in this process were: 1. Repeating the word frequency analysis across all responses to Questions A and B. This confirmed the separate institutional analyses results – that students value ‘access’ and want ‘more’ use of technologies to support their learning. 2. Manually coding a 10% sample from the institution with the most respondents (approximately 500 responses to each question) in order to identify what access students most value, and what they want more of. 3. Iteratively developing text search queries that produced at least a 90% match with most common manually identified themes in the sample. 4. Using these queries to tag all the text responses for the themes it contained. 5. Identifying where themes most frequently occurred together in the same response, indicating that students were linking these themes in some way. 6. Displaying the results as models of overall responses to each of the two questions. Having built models of the overall patterns of linkages within each of the “access to” and “more of” response sets, we were able to pull out some typical examples of comments that explained the meaning of these multiple links in the students’ own words. Overall this iterative process was a way of incorporating a grounded and reflexive approach to analysing the qualitative data; making explicit at each stage the coding rules and selection of themes for further analysis, so that they can be examined for validity. Figure 1. Content analysis of student responses – overview of methods Australasian Journal of Educational Technology, 2014, 30(1). 5 Findings The most significant findings for informing learning technology practice and policy are the themes identified in student responses, and the evidence of how these are linked – the outcomes of stages 4 and 6 in Figure 1. The most frequent word in responses to Question A was “access”. In Question B responses the most frequent word was “more” (see Figure 2). So the thematic analyses focused on finding out more about what kinds of “access” students value and what they want “more” of. Figure 2. Tag cloud displays of word frequency in responses to Question A (left) and Question B (right) Question A: Access to… The type of access students value is shown in Table 2, which lists the themes identified along with typical examples. Figure 3 presents these themes in a map, with the arrows indicating pairs of themes that are most frequently occurring in the same response. The numbered arrows indicate the top third of most frequently occurring linked pairs. In other words, the map shows where many students have linked particular pairs of themes. The findings provide evidence that the LMS plays a central role in providing access to study materials, lecture-related information and communication and that students value the flexibility and convenience of this access. Campus computing facilities, although frequently mentioned, are not so closely linked to this access. Because of the considerable investment made by universities into learning management systems we were interested to find out what attributes of the LMS were proving most valuable to students. In the model developed from Question A responses (Figure 3), the LMS is linked to “information”, “communication”, “lectures” and “study materials”. So to gain some insight into what these links are, we ran a query to pull out responses that included all four themes. Two typical examples are: [LMS] is especially useful, in particular having learning materials available online (i.e. readings, unit outline, lectures) and being able to communicate with the lecturer and other students through the discussion boards. [LMS] is a great tool for communicating with students and instructors and I feel it has assisted with my learning in a significant way. The fact that lecture notes, recordings and assessment information can be accessed at any time is a great tool … These comments highlight the assistance to learning through “anytime” access to study materials, the connectivity with staff and students to share ideas, ask questions and discuss assignments and the ability to check and find information. Students also refer to the “ease” of access. The results of the survey are interesting when considered in relation to comments in the literature about the potential demise of the LMS. Sclater (2008, p. 3) for instance argues that: ...students will find LMSs and the tools within them inferior to those they are already using freely on the internet – both in their look and feel and in the amount of functionality offered. Australasian Journal of Educational Technology, 2014, 30(1). 6 Table 2 Themes in responses to Question A (how technology helps learning), in order of decreasing frequency Theme Definition Typical examples study materials Access to digital content directly related to study Provision of PowerPoint presentations and lecture notes online … Having readily- accessible notes is important to me. information Access to administrative information such as schedules and announcements the [LMS] is crucial to my learning as it has the unit outlines and important notices LMS Any reference to the institutional online learning management system as the access point. So far I have studied approximately 50% of units externally. [LMS] (where adopted) has been a good all hours access point. convenience References to the ease or convenience of online access … I would have struggled if I did not have access to everything I need so easily. lectures Online access to lecture-related materials – notes, slides, recordings, etc. Being able to download lecture notes and tutorial activities prior to the class in order to look over it and have an idea about what is being discussed in the class. Being able to download lectures and listen, pause and rewind is good due to my ADHD assessment Access to assignment information and/or online submission and marking of assessments Being able to submit assessments online without having to waste paper. flexibility Anytime, anywhere, access to online learning Instant and available 24hr. As a full time student and work full time. my time is precious AND becomes hard to see teachers and lecturers when everyone works 9-5pm communication Use of online communication and collaboration to enhance learning … collaborate with my teachers and other students, allowing online work where I can view other students thinking and understandings which enhances my own knowledge. campus computers Computing facilities provided on campus (e.g. in labs or in the library study areas). Easy access to computers and internet on campus, which is at quick speeds and is reliable, most of the time. Statements like this have led to claims that the future of the LMS is in doubt. In the UK, Weller (2007) blogged that he thought that the “VLE/LMS is dead”. He has continued to advocate for an increasing move towards more de-centralised learning environments (Weller, 2010). However several of the comments responding to the 2007 blog article suggest caution when considering the role of the LMS and argue that there are insitutional advantages to LMSs that are not always acknowledged, such as their value as a consistent institutional interface and their ability to track and monitor via systems that are institutionally managed. The student responses in the SEET survey certainly support this note of caution because the features that students like about the LMS are currently not as easily available through a collection of de-centralised learning environments, although they may be in the future. Australasian Journal of Educational Technology, 2014, 30(1). 7 Figure 3. NVIVO model of responses to Question A. Numbers on arrows show how many students linked the themes. Question B: More of … The changes to university infrastructure that students would like to better support their learning are shown in Table 3 which lists the themes identified with typical examples. Figure 4 maps the links between these, with the arrows indicating the pairs of themes most frequently occurring together (again showing only the top third in frequency). The model in Figure 4 shows two distinct clusters of linked themes; one around how teachers are using digital media and the LMS and another around institutional IT infrastructure and hardware. This clustering shows how the institution-led and academic-led technology identified in the quantitative analysis interact to shape the student experience. Students are clearly keen for their universities to provide more facilities and infrastructure, in particular campus computer facilities, Wi-Fi and support for (student- led) use of mobile technologies. But they are also asking for their teachers to make more use of the available (institution-led) technologies, in particular LMS tools and lecture recording systems. The theme that students mostly commonly link to the LMS is teacher use of technology – with 332 responses containing both. Since this is too many to look at as a whole, a further manual coding was conducted to identify patterns. Two sub-themes that stood out as relevant in relation to student perceptions of the LMS were: i. the tools and facilities available in the LMS; ii. how teachers are using the LMS. The original query on teacher use of technologies included any reference to teacher-led online activities such as discussion, quiz, assignments etc,. This query also picked up requests for different types of online interaction than were available in the current LMS. If students wanted to use tools outside the LMS for these activities, they would be included in the query results from this theme. Less than a third of responses (100) asked for better LMS tools. The specific tools requested are clearly influenced by students’ use of similar tools in other contexts, and include: • Automatic notifications pushed to students via email or SMS (25 responses); • Mobile applications to allow student access from smart phones and tablets (20 responses); • Enhanced collaboration tools such as blogs, wikis, etc. (16 responses); • Integrated interfaces or mash-ups (only 6 responses). Australasian Journal of Educational Technology, 2014, 30(1). 8 Table 3 Themes in responses to Question B, on ways in which the University could use technology to better support learning, listed in order of decreasing frequency Themes Definition Typical examples campus computers Computing facilities provided on campus (e.g. in labs or in the library study areas). We really need more available computers especially on the [… ] campus. Alternately, making space available at various locations where students can comfortably and quietly access the internet and other online resources. teacher use of technologies How teachers are using, or not using, the available technologies perhaps lecturers can be interactive through [LMS]. It helps if the teaching staff can provide more and up to date information on [LMS]. More involvement by lecturers and faculty heads on the [LMS] IT infrastructure Technical aspects of university technology provision, such as speed of connection and reliability, and IT helpdesk support. More power points for laptops in the lecture theatres, around campus and in the library. Wider support from IT staff - I had to find out from a fellow student how to set up wireless for my Mac, because it was too new for the IT Helpdesk. Increase the internet speed! Wireless Wireless internet availability on campus Improve the wireless internet. Enable wireless access to internet via smart phones or PDAs lecture recording or podcasts Availability of lecture recordings or other audio-visual resources Recorded lectures available on [LMS]!! For all subjects!!! It would be fabulous if I could get recordings of lectures for when I am unable to get study leave from work. I would really like there to be a resource where I can listen to lectures online. There is none of this for any of my subjects and I feel that when I am sick or unable to attend uni I am disadvantaged. LMS Any reference to use of the institutional online learning management system. Adoption of [LMS] by all units and lectures. A more sophisticated [LMS] Having … email & mySR easily accessible in [LMS] that is, not having to click several times and login & out to those systems all the time. mobile technology Comments on use of mobile technologies such as smartphones and tablets Lack of support for university wifi on mobile devices e.g. iPhone etc. Introduce new technologies such as the iPad, as test experiments to select students to evaluate or create a student only price for the iPad as an incentive to students to purchase it over a notebook/laptop or netbook. Australasian Journal of Educational Technology, 2014, 30(1). 9 Figure 4. NVIVO model from B responses Over two-thirds of the linked responses asked for better teacher use of the existing LMS tools (Figure 4). This is consistent with the quantitative results, in which students’ preferred use of LMS tools is consistently and significantly higher than the current use (Gosper et al., 2013, Figure 4). Most of the comments mention basic functions such as online assignment submission and feedback, teacher presence in discussion forums, quizzes and uploading of learning resources. A number of these comments also point to a need for better institutional support for academic staff who struggle with the technology, for example: Give more incentive to academic staff to have a strong online presence. Ensuring that teachers fully understand how to use and have adequate ongoing support would aid in preventing the situation of some teachers avoiding the technology provided by the university. Often some teachers are confused as to how to activate the [LMS] site at the beginning of term (which is very frustrating), or avoid uploading things onto or communicating via [LMS] as they are not “of the technological age”. Some lecturers are unable to deal with technology at all, which has meant that hard copies of assessment tasks needed to be delivered to the university rather than being submitted electronically, and there was little interaction via electronic means. These aspects are vital to ensuring success in the delivery of learning on-line. In some cases, it might mean that a course convenor needs assistance from a colleague or PhD student to manage the technology. …train teachers on how to use the full capability of the [LMS] system. Change uni policies to be more embracing of the [LMS] system some lecturers are very reluctant for basic things such as uni assignment submission and my grades. This analysis therefore provides evidence that we need to do much more than just improve the technological tools and facilities at institutional level. Students expect their teachers to be in the virtual learning spaces with them. Many of these comments contain phrases like “some teachers” or “some lecturers”, indicating inconsistent use of the technologies among teaching staff. The 2012 ECAR study (Dahlstrom, 2012) shows a steady improvement since 2010 in undergraduate students’ perception of the effectiveness of their teachers’ use of technology internationally (but primarily in the USA and Canada). However, improving teachers’ use of these tools also involves changing the organisational systems that support learning and teaching (Russell, 2009), which vary internationally and Australasian Journal of Educational Technology, 2014, 30(1). 10 between universities. The next section of this paper therefore discusses initiatives since 2011 at the three Australian universities in the study, in the light of the findings. Institutional actions and implications The feedback from the 2010 survey was made available to each university early in 2011. At each university the data has been used for institutional planning and for informing the academic community although they used the data in different ways to inform their educational technology support strategies. Some of the infrastructure issues have been more readily addressed whereas some of the issues that emerged particularly from the qualitative data have been more difficult to tackle, in particular organisational support systems and staff development to ensure that all academic teaching staff can use learning technology effectively with their students. In this section of the paper, we review some of the changes that have taken place in relation to the findings, and draw out some implications. Institutional Planning and Policy The intention of the survey was to provide evidence that could be used for institutional planning and policy. Each university produced separate reports on the data tailored to the context of each university. These summary reports were disseminated and presented to committees such as planning and quality, IT planning committees and education committees. The reports were also circulated to Deans to inform discipline level developments. The findings have been used in various ways, of which the most explicit has been use of statistics and trends to inform planning and development initiatives. At another level, the findings have proven to be a useful reference point for monitoring trends and comparing findings from other institutional data sources. Examples of the different ways the information from the survey has been used to inform and determine institutional planning and development are listed below, although the approaches identified are not necessarily inclusive of all three universities: • At all three universities there has been continuing investment in upgrading the institutional LMS. The SEET survey provided evidence that students value this secure, easily available online management system; this has reinforced the need for institutional investment. Many of the tools requested by students in 2010 are now available, although not always widely used. Wi-Fi provision is being continually upgraded across all three universities. • At all three universities data from the survey has been used to dispel myths about the low levels of ownership and access to technology for lower socio-economic status and Indigenous students. The data showed that these groups actually had a slightly higher (not statistically significant) ownership and access to technology than the average. • Each of the three universities is implementing different learning and teaching strategies with investments in more formal planning and resourcing for blended learning. In one university, e- learning support services are being reconfigured into a more distributed model that encourages later adopters of learning technology. In another university, the move to a new LMS was seen as an opportunity to encourage transformation in learning and teaching. As part of this initiative, faculty-based support initiatives were put in place to build technical and pedagogical capability. In a third university, learning technology planning is integrated with planning for new learning experiences and new learning spaces, with the study confirming the value of the LMS for learning, along with student desires for both face-to-face and online interaction. • At each university different and specific information has been used to inform institutional changes. For example, at one university, the survey comments, along with other sources of evidence such as discussions with student representatives, provided large-scale confirmation about the value of lecture recordings. This has been taken on board and the university is now moving towards an ‘opt-out’ policy on use of the automated system for recording sound and visual materials in lecture theatres, as part of a wider investment in expanding blended learning experiences. The same (suburban multi-campus) university is also investing in the provision of mobile devices to staff and students, to enhance flexibility of access to learning resources. Another of the universities in the study is responding to the same demand by encouraging more Australasian Journal of Educational Technology, 2014, 30(1). 11 ‘flipped’ learning, in which some traditional lectures are replaced by podcasts and other content materials with face-to-face time used for collaborative learning. Curriculum development The survey also gathered data on students’ discipline and year of study, so that the student responses could be related directly to disciplinary approaches to curriculum design. In each of the universities, there is a move to integrate technology design and development with curriculum development at the program level. This removes some pressure on individual teachers to take the initiative. Teaching staff have been interested in understanding more about their students’ views on the value of learning technologies, even when they actually provide evidence that disrupts former opinions. The survey also confirmed the value that students place on “traditional” means of online communication with teaching staff and other students, such as email and LMS discussions, as well as the value of face-to-face interaction. In one of the universities, the SEET survey results provided evidence for more use of online technologies as part of curriculum redesign. In another, it confirmed the importance of developing minimum standards for online units. These standards capture students’ desire for consistent access to course information, lectures and study materials as well as effective communication between and amongst staff and students. While the survey evidence was only one of several influences on development of institutional policies, the direct influence of the survey evidence is easier to trace in specific curriculum development projects, where reports were compiled for relevant subsets of the student comments for a cohort of interest and/or on a topic of concern. For example, in one discipline, some academics argued that their low socio- economic status (SES) students would have less access to technology, citing equity as reason for avoiding online activities. Analysis of the survey data showed that low SES students in that discipline were no different from other students either in their access to technology or in their requests for more use of technology by teachers. In fact, the data from this group showed greater need for the flexibility afforded by online learning, as many are working to support their study. Another discipline was redesigning curricula to replace large lecture classes with online activities and resources combined with small group tutorial and practical classes. Discipline-specific selections of student comments provided some insight into how students might respond to this change, and reassured teaching staff that students were ready for it. Professional development for teaching staff Both the quantitative and the qualitative results from the survey provided clear evidence that students think that their teachers could be using learning technologies more effectively. There is international recognition of the need to provide more opportunities for teaching staff to improve their skills in using technology for learning. The NMC Horizon Report outlines the challenges facing institutions implementing digital technologies: As lecturers and professors begin to realize that they are limiting their students by not helping them to develop and use digital media literacy skills across the curriculum, the lack of formal training is being offset through professional development or informal learning, but we are far from seeing digital media literacy as an expected norm for academic professionals, nor as a key part of degree programs. (Johnson, Adams, & Cummins, 2012, p. 6) In the recent “Technology Outlook for Australian Tertiary Education 2013-2018” report, which draws on the views of 136 acknowledged experts in the field, the authors identify that one of the key challenges facing technology adoption is that: Most academics aren’t using new and compelling technologies for learning and teaching, nor for organising their research. (Johnson, Adams Becker, Cummins, & Freeman, 2013, p. 3). Responses from students in the SEET survey certainly reinforce these views. The responses to Question B, across all three institutions, indicate that students’ experiences of online learning are inconsistent. Australasian Journal of Educational Technology, 2014, 30(1). 12 Some teachers are using online learning effectively and raising student expectations, while others are not. One possible explanation is that e-learning support is mainly provided for individual teachers rather than at the program and department level. In the past there has been relatively limited support for educational technology embedded within disciplinary curriculum development support processes; more often it was left to the interest of teachers to develop skills. This model of support works well for early adopters (Rogers, 2003) who are actively looking for ways to use technology to enhance their teaching. However, it leaves the “laggards” free to continue their current practices unless there is a more systemic approach that involves all teaching staff, and addresses the reasons why some are reluctant to put time and effort into adopting new methods. The analysis of student comments therefore points to a need to embed support for academics’ use of online learning tools in curriculum development and teaching practice. This requires teamwork within disciplines to engage all teaching staff (academics and others who support student learning) in developing shared strategies for the use of educational technologies. A review of institutional strategy and policy changes shows that some moves are already being made in this direction in the three institutions in the study. And the analysis of student comments confirms that these changes are much needed. Some of the combined curriculum and staff development strategies introduced at the three universities are: • All three universities have increased the provision of blended learning advisors/educational developers to work with faculty staff and on specific curriculum development projects. This more distributed model offers a wide range of professional development opportunities. Examples include faculty-based plans for support activities; exemplar units, customised training, drop-in clinics, online resources, eLearning champions, eLearning networks and monthly events for staff to exchange ideas. • Curriculum development projects offer academics opportunities to work with support staff to introduce new blended learning activities. In one case, a program offered in on-campus and distance mode was redesigned to give distance students a more interactive experience, comparable with classroom activities. This encouraged staff to engage with and improve their confidence in using a range of learning technologies, and these academics are continuing to innovate. Similarly, another (fully campus-based) program is now redesigned with online activities replacing large lectures, and only small group tutorial and practical classes combined with professional practice placements. Again, teaching teams are building their capacity for innovative blended learning. • A growing range of staff development resources and events supporting blended learning, including online modules, webinars; so that all staff, anywhere, anytime, casual or full-time, can build their skills in using learning technologies in their teaching. One of the challenges in professional development for university teaching is the traditional view that individual academics have control over what and how they teach. Resistance to use of technology is sometimes linked to this perceived control, especially when the new methods are being advocated by someone else, and when students increasingly can access content not provided or mediated by the teacher (Hanson, 2009). Some of this academic individualism may be influenced by cultural differences, and Australian higher education culture has been characterised as more individualistic than some others (Moss, Kubacki, Hersh, & Gunn, 2007). Hanson (2009) suggests that resistance could also come from a strong desire of academics to protect their face-to-face relationship with their students as this has become a powerful feature of academic identity. The survey evidence does not help in addressing these issues other than to remind staff what students prefer in their learning environment. Nor does it deal with some of the systemic organisational factors, such as the need to change the way teaching work is defined, planned and measured in a traditional campus university environment. Nevertheless, the universities in this study are making some changes to address student concerns about their teachers’ capacity to use learning technologies effectively. Further comparative surveys would show whether there are changes of the scale indicated in the quantitative 2012 ECAR survey data from North America (Dahlstrom, 2012). The student voice is not the only driver of change but it can be an influential one. Australasian Journal of Educational Technology, 2014, 30(1). 13 Conclusion This study has demonstrated the value of including the student voice in developing universities’ learning technology policies and practices. In particular, the large-scale analysis of qualitative responses undertaken provides further insight into how policy and practice are affecting the student learning experience. Improving the tools and facilities at an institutional level is only part of the solution. Institutions are not going to realise benefits from their considerable investment in improved online tools and facilities unless they are also investing in complementary improvements in the organisational support for academic use of these facilities. Laurillard (2002) pointed out that for many academic staff, the introduction of new technology has been “a nightmare of overwork and lack of support” (p. 229). Our analysis of Australian university student comments in 2010 indicates that this situation may not have changed significantly. The review of changes in the three universities in this study since 2010 has shown less change in support for academic-led technology use than in deployment of institutional technical systems. While the institutions in the study are taking some actions to remedy this, our analysis provides clear evidence to support a more systemic approach to embedding learning technology in institutional curriculum development policy and practices. Students are relatively happy with the institutional systems and tools, but they want their teachers to use them more comprehensively and consistently. University staff development and curriculum development systems are both constrained by factors that may be impossible to control or predict – such as funding regimes, political priorities and demography. Because of this, teacher use of technology is harder to deal with than IT infrastructure and technical upgrades to the university online learning management system. The SEET study shows that the student voice, not only provides a quantitative indicator of progress in better use of educational technology in universities, it also provides a deeper qualitative analysis to guide the academic support strategies required to achieve that progress. On the basis of this study, we advocate a strong whole of institution approach – not just for infrastructure but also for academic leadership cascading down to individual academic teaching staff and the organisational systems that support their work. The examples of research-informed change described here show that it is at least possible to begin to make changes and that the student voice provides useful insights to guide both planning and policy directions. Acknowledgements We would like to acknowledge Dr Peter Langford from Voice Project for his advice on the development of the survey and for the survey delivery at each of the Universities. References Ackroyd, S. (2009). Research designs for realist research. In Buchanan (Ed.), The Sage Handbook of Organizational Resarch Methods (pp. 532-548; 531). London: Sage. Alvesson, M., & Sköldberg, K. (2000). Reflexive methodology new vistas for qualitative research. London: SAGE. Creswell, J. W. (2009) Research Design: quantitative, qualitative and mixed methods approaches (3rd ed.). Thousand Oaks, CA: Sage. Creswell, J. W., & Plano-Clark, V. L. (2011). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage. Dahlstrom, E. (2012). ECAR Study of Undergraduate Students and Information Technology (Research Report). Louisville: EDUCAUSE Center for Applied Research. EDUCAUSE Center for Analysis and Research. (2009) Students and Information Technology in Higher Education: 2010 Survey Questionnaire (Survey Instrument). Boulder, CO: EDUCAUSE Center for Analysis and Research, available from http://www.educause.edu/ecar. Australasian Journal of Educational Technology, 2014, 30(1). 14 EDUCAUSE Center for Analysis and Research. (2010) Students and Information Technology in Higher Education: 2010 Survey Questionnaire (Survey Instrument). Boulder, CO: EDUCAUSE Center for Analysis and Research, available from http://www.educause.edu/ecar. Ed Tech Now. (2012). MOOCS and other ed-tech bubbles. Retrieved from http://edtechnow.net/2012/12/29/moocs-and-other-ed-tech-bubbles Ernst & Young. (2011). Higher education and the power of choice: Reform, competition and the emerging consumer-driven market in Australian higher education. Retrieved from http://www.ey.com/Publication/vwLUAssets/Higher_education_and_the_power_of_choice_Australia/ $File/Higher%20education%20and%20the%20power%20of%20choice%20Australia.pdf Ernst & Young. (2012). University of the future: A thousand year old industry on the cusp of profound change. Retrieved from http://www.ey.com/Publication/vwLUAssets/Higher_education_and_the_power_of_choice_Australia/ $File/Higher%20education%20and%20the%20power%20of%20choice%20Australia.pdf Forsyth, H., Pizzica, J., Laxton, R., & Mahony, M. J. (2010). Distance education in an era of eLearning: challenges and opportunities for a campus-focused institution. Higher Education Research & Development, 29(1), 15-28. Franzosi, R. (2004). Content analysis. In M. S. Lewis-Beck, A. Bryman & T. F. Liao (Eds.), The SAGE Encyclopedia of Social Science Research Methods, pp. 187-191, Sage. Gosper, M., Malfroy, J., & McKenzie, J. (2013). Students' experiences and expectations of technologies: An Australian study designed to inform planning and development decisions. Australasian Journal of Educational Technology, 29(2), 268-282. Hanson, J. (2009). Displaced but not replaced: The impact of e-learning on academic identities in higher education. Teaching in Higher Education, 14(5), 553–64. Ipsos MORI. (2008). Great expectations of ICT: How higher education institutions are measuring up. Joint Information Systems Committee (JISC). Available from http://www.jisc.ac.uk/publications/research/2008/greatexpectations.aspx JISC. (2011). Student Experiences of Technology. Retrieved June 11, 2011 from http://www.jisc.ac.uk/whatwedo/campaigns/studentexperiences.aspx Johnson, L., Adams, S., & Cummins, M. (2012). The NMC Horizon Report: 2012 Higher Education Edition. Austin, Texas: The New Media Consortium. Johnson, L., Adams Becker, S., Cummins, M., & Freeman, A. (2013). Technology outlook for Australian tertiary education 2013-2018: An NMC Horizon Project regional analysis. Austin, Texas: The New Media Consortium. Kennedy, G., Judd, T., Churchward, A., Gray, K. & Krause, K. (2008). First year students’ experiences with technology: are they really digital natives? Australasian Journal of Educational Technology, 24(1), 108-122. Laurillard, D. (2002). Rethinking university teaching: A conversational framework for the effective Use of Learning Technologies. London: Routledge Falmer. Little, B. Locke, W, Scesa, A., & Williams, R. (2009). Report to HEFCE on student engagement. Centre for Higher Education Research and Information, The Open University. Retrieved from http://www.hefce.ac.uk/media/hefce/content/pubs/2009/rd0309/rd03_09.pdf Moss, G., Kubacki, K., Hersh, M., & Gunn, R. O. D. (2007). Knowledge management in higher education: A comparison of individualistic and collectivist cultures. European Journal of Education, 42(3), 377-394. Australasian Journal of Educational Technology, 2014, 30(1). 15 Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York: Simon & Schuster. Russell, C. (2009). A systemic framework for managing e-learning adoption in campus universities: individual strategies in an institutional context. ALT-J Research in Learning Technology, 17(1), 3-19. Sclater, N. (2008). Web 2.0, personal learning environments, and the future of learning management systems. ECAR Research Bulletin, 2008(13). Retrieved from http://www.educause.edu/library/resources/web-20-personal-learning-environments-and-future- learning-management-systems Weller, M. (2007). The Ed Techie: “The VLE/LMS is dead” [Web log post]. Retrieved from http://nogoodreason.typepad.co.uk/no_good_reason/2007/11/the-vlelms-is-d.html Weller, M. (2010). The centralisation dilemma in educational IT. International Journal of Virtual and Personal learning Environments, 1(1) 1-9, January-March. Corresponding author: Carol Russell, c.russell@uws.edu.au. Australasian Journal of Educational Technology © 2014. Please cite as: Russell, C., Malfroy, J., Gosper, M., & McKenzie, J. (2014). Using research to inform learning technology practice and policy: a qualitative analysis of student perspectives. Australasian Journal of Educational Technology, 30(1), 1-15.