Australasian Journal of Educational Technology, 2014, 30(6). Transforming higher education and student engagement through collaborative review to inform educational design Brian R. von Konsky, Romana Martin, Susan Bolt, Tania Broadley Curtin University, Australia Nathaniel Ostashewski Athabasca University, Canada This paper reports on staff perceptions arising from a review process designed to assist staff in making informed decisions regarding educational design, approaches to engage students in learning, and the technology to support engagement in the classroom and across multiple locations and delivery modes. The aim of the review process was to transform the level of student engagement in the business faculty of an Australian university. The process took a collaborative approach through consultation with academic staff involved in the design and delivery of the units under review, and included targeted professional development as necessary. An institutional framework that characterises engagement indicator contexts and their attributes facilitated dialog during the review process. This paper reports on a mixed method study that included a survey of participants, and purposeful interviews to evaluate the effectiveness of the process. Although the study identified factors that hindered implementation and operationalization of review recommendations in some instances, study participants were generally of the view that recommendations would enhance student engagement. It is demonstrated that the bottom-up approach described in this paper is consistent with theoretical frameworks for transformational change in teaching and learning and the adoption of innovations. Introduction Students are said to be engaged in learning when they actively participate in challenging learning activities, particularly those that develop higher order thinking skills involving interaction and collaboration with peers (National Survey of Student Engagement, 2012). Active and problem-based learning are two such approaches. Active learning is an approach to educational design that is based on activities in which students work collaboratively on tasks that contribute to their learning (Chickering & Ehrmann, 1996; Chickering & Gamson, 1987; Meyers & Jones, 1993). In Problem-based Learning (PBL) student teams develop analytical skills and identify what they know, what they need to know and how they will acquire this knowledge, and develop a plan to collaboratively work towards the solution of a stated problem (Tight, Mok, Huisman, & Morphew, 2009). Active learning and PBL problems that take place in face-to-face settings can also be reconceptualised for implementation in blended and fully online settings (Dziuban, Hartman, & Moskal, 2004). Moreover, there is a growing body of evidence that online technologies can be used to enhance student engagement and lead to improved learning outcomes when used as part of an effective educational design (Coates, 2007; US Department of Education, 2010). Staff capacity to effectively engage students in online learning environments is essential because in the current climate of rapid change, university students are demanding flexibility in learning. This has led to considerable growth in online enrolments worldwide. In the United States alone, online enrolments expressed as a percentage of total enrolments have grown from 9.6% in 2002 to 31.3% in 2010 (Allen & Seaman, 2011). Coupled with the additional economic affordances of technology, this growth has led some institutions to regard online learning as a strategic asset (McCarthy & Samors, 2009). However, academic staff do not necessarily share the view of university administrators regarding the strategic importance or educational efficacy of online learning (Seaman, 2009). For example, only 20% of university administrators recently reported having more fear than excitement regarding the growth of online education (Allen, Seaman, Lederman, & Jaschik, 2012). Of these, around 5% of administrators were of the view that learning outcomes arising from online study modes were inferior to those arising from face-to-face instruction. In sharp contrast, almost 60% of teaching staff reported feeling more fear than excitement, and as many as 30% were of the view that learning outcomes for online study modes were inferior to face-to-face modes. This was despite evidence reported in the literature demonstrating the efficacy of online approaches for improving student engagement and learning outcomes (US Department 619 Australasian Journal of Educational Technology, 2014, 30(6). of Education, 2010). However, the literature shows that the attitudes and the role of the teaching academic in encouraging student engagement are critical (Zepke & Leach, 2010). For instance, Bryson and Hand (2007) found that the levels of enthusiasm, discourse between staff and students, and the professionalism demonstrated by academic staff impacted on students’ engagement in learning in a university business course. To address the disparity in staff perceptions of online learning and the need for more engaging online teaching, professional development is an important strategy used by many institutions to enhance academic staff skills and improve confidence regarding the use of online technologies in teaching and learning (Singh, Schrape, & Kelly, 2012). This can include professional learning that spans an extended period of time. For example, Donnelly (2010) describes using a problem-based learning approach in a blended professional development program for teachers, spanning a period of 10 weeks. In part, this was intended to model engagement strategies in a more authentic manner (Zepke & Leach, 2010). Despite successes, there is evidence that formal professional development alone is insufficient to change the beliefs and attitudes of teaching staff (Guskey, 2002). Effective professional development requires the learning to occur within meaningful, relevant contexts. This includes learning on-the-job in response to current issues that necessitate immediate action and enables teachers to collaborate within communities of practice (Bolt, 2012; Knowles, Holton, & Swanson, 2005; Lloyd & Duncan-Howell, 2010; Zepeda, 2012). To bring about change resulting in professional growth, teachers need opportunities to engage with relevant theory and time to transfer their learning by putting it into practice within their teaching contexts (Bolt, 2012; Caffarella, 2002; Lloyd & Duncan-Howell, 2010). Such transformational change can often be facilitated through mentoring and coaching relationships between teachers and more experienced colleagues (Zepeda, 2012). Additionally, the adoption of innovations can be facilitated by early adopters, who contribute to change through leadership and example, modelling new practices to those less inclined to change (Rogers, 2003). Many scales have been designed to measure engagement from the student experience perspective (Oliver, Tucker, Gupta, & Yeo, 2008; Robinson & Hullinger, 2008). For instance, Chao, Saj, and Tessier (2006) employed a framework for managing the quality of online teaching and learning. They reported on a pilot study to consider framework components using a review based on an institutional style guide for online courses. In particular, the review considered the instructional design, web design, and presentation for each unit in the pilot. This included the extent to which online learning activities made effective use of technologies that were aligned with the intended learning outcomes. The review also considered the online presence of each unit with respect to site usability, consistency, grammar, and look and feel. The review used a four-point scale to determine if each category was satisfactory or in need of improvements that were discretionary, minor, or significant in nature. In this instance, the style guide served as the basis of the review used for the quality management of online course offerings. Chao and colleagues (2006) further suggest that a style guide serves as a checklist to assist staff in the development of online units based on an established set of institutional expectations. Other similar institutional frameworks and checklists have been proposed for the review of online units. For example, Edith Cowan University has developed quality guidelines for online courses and a checklist for assessing the quality of online units (Herrington, Herrington, Oliver, Stoney, & Willis, 2001). Their approach identifies pedagogies with authentic tasks, collaboration opportunities, and meaningful assessment; the inclusion of learning resources that are current, inclusive, accessible, and with appropriate media; and robust online delivery that considers bandwidth constraints and adheres to a consistent presentation style and format. This approach was primarily used for auditing compliance with institutional guidelines and identifying areas requiring improvement. The work described in this paper shifts the focus of online student engagement review from audit and compliance to building staff capacity, confidence, and skills. The focus of the study reported in this paper is on the utilisation of a framework designed to enable teachers to reflect and critically analyse their approaches to designing learning activities for online and blended learning environments. The process is based on a collegiate and collaborative approach in which teaching and learning specialists work with small groups of lecturers who teach related subjects. The review process documents current practices for engaging students and results in recommendations to raise the level of student engagement in individual subjects and across majors as a whole. Perhaps more significantly, the process has led to targeted 620 Australasian Journal of Educational Technology, 2014, 30(6). professional development based on identified needs. Equally significant, it has led to collaborative teaching and learning research based on identified opportunities. This paper reports on results arising from that process, assessed in the year following the completion of the reviews. These results include the extent to which staff adopted the recommendations, the extent to which they found them to be useful, and factors that were barriers to their adoption. Institutional Setting This study was conducted in a business faculty of a large Australian university using a framework providing guidelines for fostering student engagement in blended learning environments that was developed at the authors’ institution in 2011. In accordance with institutional directions, the authors’ view of engagement was aligned to this framework. In the context of teaching higher education students in blended learning environments, the framework referenced five indicators that influenced student engagement (Curtin University, 2011). These indicators were largely based on accepted definitions of student engagement (Coates, 2007) and were similar to those that have been used elsewhere (National Survey of Student Engagement, 2012). The indicators in the framework were: learning resources; learning activities; communication and collaboration; student support; and assessment and feedback. The framework unpacked each of these indicators by describing teaching and learning scenarios across three contexts indicative of diverse approaches and non-scaffolded levels of implementation of effective strategies to engage students in blended learning environments. An excerpt of the framework is shown in Table 1 to exemplify the way in which indicators of student engagement (for example, learning activities) were elaborated upon by describing teaching and learning scenarios across three Contexts. At a minimum, all units were expected to demonstrate Context 1 attributes across each of the five engagement indicators within a learning environment that used the institutional Learning Management System (LMS) to organise and distribute learning material; learning resources that contained well stated learning outcomes aligned with assessment; activities with clearly stated participation expectations; and student support facilitated by accessible academic staff and through links to institutional resources like library tutorials and referencing styles. Table 1 The attributes for the learning activities indicator L ea rn in g A ct iv it ie s Context 1 Context 2 Context 3 • Clearly stated expectations of student participation • Activities align with unit outcomes and assessment • Instructions and feedback on satisfactory completion of learning activities • Activities that facilitate student engagement (e.g. blogs, wikis, journals) • Learning activities are authentic • Online activities to support independent learning • Scaffolded activities culminating in a final product (e.g. website, performance, demonstration) • Student centred learning tasks that extend student engagement and collaboration (e.g. creation of digital interviews, peer-review, digital mash-ups) • Learning tasks have depth, complexity, and duration • Problem-based learning • Opportunity for self- directed learning Contexts 2 and 3 extend Context 1 to describe attributes that enhance student engagement across each indicator, using the affordances of technology as appropriate. For example, Table 1 shows attributes for the three contexts for the learning activities indicator. Context 2 activities are generally those that are authentic, supporting independent learning in order to produce a final product. Context 3 extends this to include tasks that have depth and complexity, with significant opportunities for both collaboration and self-directed learning as appropriate. The complete framework showing all engagement indicators is available online (Curtin University, 2011). The framework was used to inform a collaborative review process of majors offered in conjunction with the Bachelor of Commerce course. The review process was designed to: document current practices that positively influence student engagement; identify and disseminate examples of best practice; recommend 621 Australasian Journal of Educational Technology, 2014, 30(6). pedagogies and technologies to enhance student engagement; and to inform targeted professional development based on identified needs. Staff associated with the teaching and learning unit embedded within the business faculty conducted the reviews. This team worked in close collaboration with Unit Coordinators and teaching staff associated with the majors under review. At the time of the review, subjects were offered both onshore and in a number of offshore locations. They were also offered in a variety of delivery modes, including blended mode and in some cases through distance education. A total of seven majors and all common units in the first year were reviewed in a period spanning late 2011 through 2012. Five of the majors were in the process of being re- conceptualised for fully online implementation. An additional two majors were under consideration for fully online deployment at the time of the review. In all cases, a goal of the review was to adopt a common educational design across all modes and locations, taking advantage of the affordances of technology to suit the learning needs of the each cohort in a sustainable and engaging fashion. The process involved: (1) meeting with individual heads of school and school-based directors of teaching and learning to seek their feedback on strategic engagement priorities specific to their areas; (2) a staff meeting to review the engagement process and its timeline; (3) conducting an initial needs analysis against the five engagement indicators based on available data and artefacts; (4) face-to-face meetings and email consultations to discuss the engagement indicators in the context of specific subjects, often in clustered groups; (5) developing a draft engagement report for each subject containing a list of commendations and recommendations based on the five engagement indicators; (6) meetings to discuss the draft unit reports and its commendations and recommendations; and (7) writing a final report with generalised commendations and recommendations for the major as a whole, and containing a collated version of final unit reports. Throughout the process, teaching staff were invited to attend professional development sessions based on identified needs, as offered by the university teaching and learning unit (Singh et al., 2012). These included standardised training on topics like using discussion boards, blogs, and wikis in teaching and learning. Targeted professional development was also offered within the business faculty based on specific needs identified during the reviews. For example, some subjects utilised learning resources available on the web site for the textbook associated with a given subject. This included video clips, interactive media, scaffolding and consolidation quizzes, and case studies. In some instances, this was used very effectively, with resources being linked to specific learning activities and assessments. In other instances there were many resources from a variety of sources that were not directly linked to learning activities or assessments, or there were no resources made available at all. Having identified a clear need during the review, a targeted professional development workshop was held on making effective use of publisher resources. This was based on a case study and a comprehensive environmental scan conducted by a member of the review team. This included detailed examples of best practice and an analysis of available public resource strategies (Martin, 2012). Similarly, the extent to which active learning strategies were utilised was seen to be inconsistent. In some majors, active learning was commonplace, while in other majors active learning activities were not usually employed. In some instances, this was attributed to logistical issues associated with teachers’ perceptions of implementing active learning activities in classes with large enrolments in lecture theatres designed for didactic teaching. Even so, active learning activities could be used in large lecture theatre environments (Biggs & Tang, 2011; Mazur, 2009) facilitated through the adoption of appropriate technology (Deslauriers, Schelew, & Wieman, 2011; Oblinger, 2005). Consequently, a targeted professional development session for academic staff was held to demonstrate technologies like Hot Seat, and to model its use in lecture theatre settings. Moreover, the engagement review process was in itself a form of professional development because it enabled small clusters of teaching staff and the review team to reflect on and address issues in relation to approaches to student engagement in the specific context of a small number of related subjects. It was intended that this consultative approach to professional development would enhance its relevance, and improve the sense of buy-in and local ownership. It was further intended that this would lead to the adoption of consistent interaction and participation expectations for students across related subjects. 622 Australasian Journal of Educational Technology, 2014, 30(6). Following the conclusion of the review process, teaching teams were in a position to make an informed decision about the best way to raise the level of student engagement in their subjects and majors, with subsequent follow up consultation and support available from the faculty teaching and learning specialists. Some teachers implemented many of the recommendations arising from the reviews. The extent to which this approach facilitated the adoption of new practices to enhance student engagement was not directly known, nor were factors that were barriers to the uptake of these recommendations. This observation informed the study reported in this paper. Methodology In 2013, the authors conducted a study to evaluate the results of the review they conducted in 2012 with teaching staff in a university business faculty. This study was conducted with the approval of the Human Research Ethics Committee at the authors’ institution (Approval Number: CBSFac-1-2013). The goal of this study was to inform future iterations of review processes; to build business faculty staff capacity to implement teaching and learning strategies that enhance student engagement; and inform management of future engagement projects aligned with institutional priorities and initiatives. This is the principal contribution of this paper. In a broad sense, like many social and educational research projects, this study could be described as a case study because it investigated a review process conducted within a bounded timeframe with a small number of purposive participants using mixed methods to collect and triangulate data (Creswell, 2009; Tight, 2012; Yin, 2014). Indeed, the case study described in this paper was limited to an investigation within the business faculty where the authors were employed as educational consultants at the time of the study. Consequently, within the scope of this paper, it is not possible to provide further details about the use of the review process in other contexts. In the future, other researchers may like to investigate this. The overarching research question was: How can student engagement be enhanced in higher education online and blended learning environments? The secondary research questions were: 1. What are staff perceptions and beliefs about online learning and student engagement? 2. What are the key benefits and outcomes arising from the student engagement review process? 3. What factors hindered the adoption and operationalisation of review recommendations? 4. What outside issues do staff believe adversely impact student engagement? Data collection and analysis This study was conducted at the course level (a 3 year Bachelor of Commerce degree) and data were collected at the unit level (12 week programs of study which students typically attend for 3 hours a week on a semester basis) (Tight, 2012). The authors adopted a mixed-method approach and used a concurrent triangulation strategy (Creswell, 2009). Quantitative and qualitative data were collected from an online survey, purposeful semi-structured interviews, and field notes and documentation directly arising from engagement reviews (Yin, 2014). Understandably, the field notes and documentation were collected as part of the initial review process. The online surveys were conducted prior to the semi-structured interviews for logistical rather than research purposes; hence, the research design was considered to be concurrent rather than sequential (Creswell, 2009). One of the limitations of this approach was that participants felt that they had said everything they wanted to say in the survey. Consequently, only 3 of the 10 people invited to participate in semi-structured interviews did so. A strength of this approach was triangulation of data and confirmation that saturation had been achieved. Numerical data from the survey were analysed statistically and reported in the Figures and Tables included in this paper. Open-ended questions in the survey and transcribed interview data were analysed thematically and quotations from participants are included in this paper. Participation Heads of schools, unit coordinators, and teaching staff who had previously participated in an engagement review were invited to participate in this study. As a result, a total of 51 members of the teaching staff were invited to take an anonymous online survey. Of these, 21 academics (41%) provided informed consent to participate. However, a majority of survey questions were answered by 13 (25%) to 15 (29%) 623 Australasian Journal of Educational Technology, 2014, 30(6). of survey participants. Because participants did not uniformly answer all questions in the survey, the sample size is independently shown for each question when reporting results. As noted, the sample size for individual questions generally varies between 13 and 15 responses. Despite the low number of respondents, demographic data showed that those participating in the survey represented all business disciplines reviewed during 2012, and were involved in the teaching and coordination of subjects taught in multiple modes and locations. The survey included a combination of four-point Likert scale, select all that apply, true or false, and open-ended questions. Qualitative data from open-ended questions in the surveys were supplemented with selected purposeful interviews. A purposive subset of 10 participants was invited to be interviewed. Of these, 3 people agreed to be interviewed. The interviews were recorded and transcribed for analysis purposes. The names of the review staff mentioned by participants in the transcripts were changed to maintain confidentiality, as were the names of identifiable majors and courses. Results Teaching staff who participated in the Engagement Review were given reports of findings in relation to evidence of student engagement within their units. A final overarching report arising from each Engagement Review included outcomes of the review process for majors as a whole, as well as Unit Reports for individual subjects. The dissemination of these reports to key stakeholders at the unit, school and faculty level informed participants and provided an element of validation to the research (Creswell, 2009). The extent of participants’ agreement with findings presented in the reports is shown in Figure 1. Figure 1. Extent of agreement with statements about the Engagement Review report. Most study participants agreed that the report contained an accurate description of engagement in their unit at the time of the review (strongly agree=3, agree=9, disagree=1; n=13). Most also agreed that the recommendations identified in the report were useful to the academic staff responsible for the unit (strongly agree=1, agree=10, disagree=2; n=13). There was also agreement that the report identified existing aspects of the unit that work well to engage students (strongly agree=4, agree=8, disagree=1; n=13). Indeed, the Engagement Review report documented how students were being engaged in each unit at the onset of the review process and included commendations to document examples of best practice when identified. On the basis of this, one participant cited using the Engagement Review report in a subsequent application for a teaching award: “As a by-product [of the review], the [teaching team] got a T&L award for the ‘Programs that Enhance Student Learning’. A bonus.” Of those responding (n=11), 100% indicated that recommendations in the report were likely to lead to improved levels of student engagement. Of these, 27% (n=3) felt that report recommendations were strategic in nature; 36% (n=4) felt that report recommendations were sustainable; but only 18% (n=2) felt that the recommendations were scalable with respect to cohort size. Of those responding (n=13), there were 23% (n=3) that agreed with all of the recommendations in the engagement reports; 69% (n=9) reported that they agreed with some but not all of the recommendations; and 8% (n=1) indicated that they did not agree with any of the recommendations. Although most staff 624 Australasian Journal of Educational Technology, 2014, 30(6). agreed with report recommendations to some extent, 31% (n=4) did not intend to implement recommendations because they did not have ongoing responsibility for the unit. Other barriers to the implementing Engagement Review report recommendations are shown in Table 2. In the following paragraphs, the results of this study are presented sequentially and organised thematically according the secondary research questions. Findings from quantitative and qualitative data have been integrated under these thematic headings to provide a richer understanding of these phenomena. What are staff perceptions and beliefs about online learning and student engagement? The number of people who agreed or disagreed with statements regarding their beliefs about online learning and student engagement using a four-point Likert scale is shown in Figure 2. Green shows the number of respondents who agreed with a statement. Red shows the number of respondents who disagreed with a statement. Around two-thirds of those responding to the survey agreed with the statement that they have a clear understanding of what is expected of them when teaching online (strongly agree=2, agree=8; n=15). Opinions were evenly divided regarding whether students can learn online as effectively as they can in face-to-face settings (strongly agree=1, agree=7, n=15). None the less, most were of the view that it is possible to create online learning experiences that are equitable with respect to those offered face-to-face (strongly agree=1, agree=7, n=15). Most agreed that the student engagement review process provided a forum for their views regarding online learning to be heard (strongly agree=1, agree=12; n=15). Figure 2. Extent of agreement with statements about online learning and student engagement. What are the key benefits and outcomes arising from student engagement review process? The extent of agreement with statements about the engagement review process is shown in Figure 3. Most staff reported that the time spent in the engagement review process was of value to them (strongly agree=4, agree=10, disagree=1; n=15), and resulted in approaches were likely to improve student engagement (strongly agree=1, agree=10, disagree=3, strongly disagree=1; n=15). Moreover, participants generally agreed that engagement reviews resulted in new ideas to raise student engagement that had not previously been considered (strongly agree=3, agree=9, disagree=3; n=15). That is, the engagement review introduced new approaches to assist academic staff to raise the level of student engagement and challenge existing practices. For example, one participant commented that a benefit of the engagement review process was that it “opens the mind to thinking outside the box.” Similarly, another participant indicated that a benefit was “making me think and challenge what I think.” Participants were generally in agreement that the engagement review process resulted in good working relationships with the review team (strongly agree=6, agree=7, disagree=2; n=15), the latter of which 625 Australasian Journal of Educational Technology, 2014, 30(6). provided subsequent technical and pedagogical support, targeted professional development based on needs identified during the review, and referrals to training offered by the University’s central academic development unit. Indeed, academic staff reported placing value on the knowledge, skill, and collegiate interaction of the review team. For example, one participant reported that a benefit of the process was “meeting individuals in the engagement review team who are knowledgeable about ways to engage students online and who offer valuable suggestions.” Another echoed this view, stating “the observations that Steve gave and the opportunities for improvement were I think the best thing I got from the review.” Another academic provided a specific example of one of Steve’s suggestions: I’d already used [Blackboard] Collaborate on quite an ad hoc basis, but after talking to Steve about that I realised that I could use it in different ways.” He continued by explaining that “… one of my challenges was getting the external [online] students to do all the presentations, and I was really not sure about using Collaborate for that, but after the Engagement Review that’s what I’ve done.” An increased awareness of approaches to enhance student engagement was apparent across all engagement indicators. Of those responding (n=13), a total of 69% (n=9) of survey participants reported that they were more aware of ways to enhance student engagement associated with the assessment and feedback engagement indicator as a consequence of participating in the Engagement Review; 46% (n=6) reported being more aware of factors associated with both the learning activities and the communication and collaboration engagement indicators; 31% (n=4) reported begin more aware of ways to enhance engagement through the effective use of items associated with the learning resources engagement indicator. Engagement indicators in the framework were not considered in isolation during the review. Instead, indicators were considered to be components of a holistic educational design that should be applied across all delivery modes and locations. “Learning how I can engage online students more and providing a more integrated approach to teaching [both] online and face-to-face students to the point now where I think I can almost have one Blackboard site for both sets of students” was a reported benefit of this approach. When asked if other majors and units in the business faculty would benefit from participating in an engagement review, only a single individual was of the view that it would not. The remaining 90% (n=9) indicated that a review would be valuable in other areas. Figure 3. Extent of agreement with statements about the engagement review process. What factors hindered the adoption and operationalisation of review recommendations? Factors hindering adoption and operationalisation of review recommendations indicated by survey participants are given in Table 2 (n=12). More than half of the study participants reported that insufficient time allocation in the university’s workload management system and lack of suitable administrative support hindered their ability to implement report recommendations. Half reported that lack of additional teaching staff similarly hindered implementation, along with lack of necessary infrastructure like iPads 626 Australasian Journal of Educational Technology, 2014, 30(6). and clickers. Only 25% reported that they needed additional professional development training or pedagogical support to implement recommendations. However, it is unclear if these individuals had attempted to access available professional development opportunities available through the central university academic development unit, or via the faculty-based review team. Moreover, while academic staff had access to faculty-based support and professional development opportunities, this was not available to the same extent for students. This was seen a barrier to operationalising technologies for student engagement by some teaching staff, who report spending time assisting students with technical difficulties in a manner that detracts from the learning experience. One staff member stated: Plus there’s a lot of to-ing and fro-ing for students just trying to sort out their technical issues. And it could be something as minor as the fact that they don’t realise that Internet Explorer doesn’t work well with Blackboard, or they haven’t updated their plug ins or whatever it may be, and we end up spending quite a lot of time on that. Table 2 The percentage of participants (n=12) indicating that the implementation of some report recommendations require specified additional resource or support Addition resource or support required % n Academic Workload Management System (AWMS) time-allocation 67% n=8 Administrative support 67% n=8 Technical support 58% n=7 Teaching staff 50% n=6 Technical infrastructure (e.g. iPads, clickers, microphones) 42% n=5 Professional Development 25% n=3 Pedagogical support 25% n=3 What outside issues do staff believe adversely impact student engagement? Other factors hindering student engagement as perceived by survey participants are given in Table 3 (n=14). The table indicates the perception of survey participants (n=14) that a significant outside factor influencing student engagement was the layout of face-to-face learning spaces, followed closely by technological infrastructure such as access to fixed and wireless networks. Around a third of those responding indicated that the timetabling of lecturers and tutorials and issues with online systems also had an adverse impact on student engagement. Poor attendance in synchronous online tutorials was deemed to be of greater adverse impact compared to attendance in face-to-face classes. For example, one study participant reported: In some of the units the attendance has been quite low in the [synchronous] online tutorials, and I just wonder whether we’re trying to do something that’s actually not really particularly wanted. The participant continued by adding: …. they’re across all these different time zones. So there is a challenge there. If you set a time for a tute, let’s say you’ve got two groups of students, or three groups of students, you can set three different times, but they’ll always be someone who can’t make that particular time, and so that is an issue, because then they start feeling ‘Oh, we’re missing out’, and of course it’s not centrally timetabled like our internal classes. To a great extent, this reinforces the importance of considering student needs and expectations when contextualising an educational design and selecting appropriate technologies. It also suggests the need to take resource and staffing implications into consideration. For example, hiring tutors in distributed regions rather than sourcing them from the home campus may be a better alternative for timetabling of remote synchronous learning experiences. 627 Australasian Journal of Educational Technology, 2014, 30(6). Table 3 The percentage of participants (n=14) indicating that a given outside issue adversely impacts on student engagement in a unit that they teach Issue impacting student engagement % n Layout of the learning spaces (e.g. fixed seating) 50% n=7 Technological infrastructure (e.g. network/wireless access) 43% n=6 Timetabling of lectures or tutorials 36% n=5 Other online learning systems (e.g. Echo 360, Blackboard Collaborate) 36% n=5 Attendance in online tutorials for my unit 29% n=4 The Learning Management System (LMS) e.g. Blackboard 21% n=3 Attendance in face-to-face tutorials for my unit 14% n=2 Other 7% n=1 Discussion Roughly half of the academic staff participating in this study were of the view that students cannot learn online as readily as they can in face-to-face settings. This belief is generally consistent with data reported elsewhere in the literature (Allen et al., 2012). This suggests that the student engagement review process did not significantly impact academic staff views regarding the efficacy of online learning. None the less, academic staff participating in this study were overwhelmingly of the view that the student engagement review process was of value to them; that it was likely to raise the level of student engagement in the units that they teach; and that other programs within the business faculty would benefit from undergoing the process. This apparent contrast is not entirely surprising, given that the primary focus of the engagement reviews was to raise student engagement based on the five engagement indicators in the framework, rather than on implementing online technologies as such. However, the role of technology and its many affordances was considered during the review as a means to facilitate student engagement based on a sound educational design, and as a means to provide for equitable learning opportunities across modes and availabilities. For example, during an engagement review a unit coordinator expressed frustration that a significant number of students did not attempt to answer tutorial questions prior to attending class. The unit coordinator further explained that tutors found it difficult to motivate students to participate while in class. Perhaps more to the point, there was anecdotal evidence to suggest that students were unclear about the significance of the unit with respect to their chosen business major. The engagement review of this unit showed that the primary focus of planned learning activities and assessments was solving mathematical problems from the textbook. Little attention was placed on how to apply these techniques to inform solutions to real-world issues in business. The review recommended that a case-based approach be adopted that would better position the significance of the mathematical skills in business settings. The unit report contained a reference to an article with sources containing suitable business cases (Parr & Smith, 1998). The reference also described approaches for authentic business case-based activities that enable students to apply the underlying mathematical knowledge. Adopting a case-based approach would serve to demonstrate the relevance of the underlying mathematical skills and knowledge to business. It also potentially provides for the development of other important attributes of a business professional. These include the development of critical thinking, teamwork and presentation skills. These skills are developed when case-based activities require student teams to consider a business issue, identify a mathematical approach to analyse the issue, and present cogent recommendations based on their analysis to others who do not necessarily possess the same level of mathematical ability (e.g. consumers, sales staff, the courts, the Board of Directors). The unit report recommended that case-based presentations could be done orally during face-to-face tutorials. The report further recommended that other case-based presentations could be done online using a wiki to enable students to collaborate on case studies outside of scheduled class time. Employing a blended approach has been shown to improve student learning outcomes in some instances (US Department of Education, 2010). A further benefit is that this serves as preliminary step towards implementing a fully online version of the unit. At the time of the review, the unit in this example demonstrated attributes of Context 1 across all five engagement indicators. The recommendation that the unit adopt a case-based approach was intended to transform this unit, introducing authentic activities that facilitate student engagement, and that incorporate 628 Australasian Journal of Educational Technology, 2014, 30(6). the use of wikis to promote interaction and collaboration on business cases outside of formal class meetings. Effectively, adopting the recommendation would transform the unit from one that generally only demonstrates Context 1 attributes to one that demonstrates significant attribute from Context 2 in the learning activities, communication and collaboration, and assessment engagement indicators. In most instances, unit coordinators worked in close collaboration with the faculty-based teaching and learning unit to adopt all or some of the review recommendations. In addition to including members of the review team, the faculty-based teaching and learning unit supporting the implementation and operationalisation of review recommendations included additional educational developers, instructions designers, graphics designers, and LMS specialists. In some cases, however, unit coordinators chose not adopt the recommendations of the engagement review. This study has identified a number of factors that hindered the implementation and operationalising of recommendations by some academics. Chief amongst these was a prevailing view that staff had insufficient time allocation in the University’s Academic Workload Management System (AWMS) to implement all the changes, even with the support of the faculty teaching and learning unit, or that they lacked appropriate administrative support. This finding is generally consistent with the results of an Australian Learning and Teaching Council (ALTC) funded project entitled “Planning and implementing a benefits-oriented coasts model for technology enhanced learning” (Tynan, Ryan, Hinton, & Mills, 2012). In interviews encompassing academics from a number of Australian higher education institutions, the ALTC study found that academics largely held that workload was often underestimated and sometimes failed to consider to the impact of technology on course development and preparation. Moreover, many staff in the ALTC study reported that teaching allocations based on student load were inadequate given different workflows associated with teaching online, and demands arising from student assumptions and expectations. There is also a growing body of evidence that other factors including the layout of the learning space can impact student engagement (Matthews, Andrews, & Adams, 2011). This study has also found that the layout of the learning spaces and issues with the technical infrastructure and online learning systems had a negative impact on the ability of the staff to engage students in the classroom and the operationalisation of review recommendations. Outside factors like workload and the layout of teaching spaces notwithstanding, transforming learning can require changing the beliefs, and attitudes of teaching staff who implement and operationalise teaching innovations proposed by others. A model by Guskey (2002) postulated changing beliefs and practices cannot be achieved by professional development alone. He pragmatically recognized that changes in staff beliefs and attitudes could only truly occur after new teaching practices were implemented and staff observed improvements in student learning. This observation also suggests that curriculum renewal initiatives and external reviews by themselves do not necessarily lead to transformational change that is sustainable unless teaching staff have direct evidence that implementing the change was actually of value. Guskey (2002) noted that this could include improvement in standard indicators like marks, or observations that new teaching practices led to better student behaviour, motivation, attitudes or engagement. Similarly, Rogers’ Diffusion of Innovations Theory stipulates that new innovations are only likely to be adopted if new innovations are consistent with existing values and practices, and are perceived to be better than what they replace (Rogers, 2003). Rogers also recognised that new innovations are usually not adopted at a uniform rate across a community. Some individuals are more likely to drive or adopt new innovations than others. Rogers observed that innovators and early adopters are in a strong position to influence their peers. Despite this influence, Rogers further observed that many are inclined to wait until a majority of their peers have adopted an innovation. A small number are usually consistently amongst the last of their community to adopt a new innovation. To a great extent, the engagement review process is compatible with the theoretical models of Guskey (2002) and Rogers (2003). It is a bottom-up approach to transformational change that is faculty-based. It is bottom-up in the sense that it empowers teaching staff to make informed choices about the way that they engage students in learning. Equally important, it provides them with the on-going faculty-based support to ensure that they are successful. It recognises that professional development, by itself, is 629 Australasian Journal of Educational Technology, 2014, 30(6). insufficient to transform the beliefs and attitudes of teaching staff. It further recognises that some staff are naturally inclined to adopt innovative teaching practices, whereas others are less inclined given factors that they consider to be barriers. There is significant anecdotal evidence that staff participation in scholarship of teaching and learning opportunities arising as a consequence of engagement reviews has been an agent of transformational change within the business faculty. This has included joint research projects and peer reviewed publications with members of the faculty-based review team, as well as small research projects funded at the institutional level. In that sense, the process has resulted in some staff taking a scholarly approach to determining the outcomes of their innovative teaching practices. Identifying examples of best practice in engagement reviews has also led to seminars in which innovators share their outcomes with their peers across disciplinary boundaries. In these seminars, innovators lead their peers to consider adopting innovative approaches in their own teaching. The engagement review process has also produced documented evidence to facilitate successful applications for teaching and learning awards. Many institutions have well-established initiatives for curriculum review and renewal (Oliver, 2013; Spencer, Riddle, & Knewstubb, 2011), updating teaching spaces and related technology (Mathews, Andrews, & Adams, 2011), work allocation, and quality management (Oliver, Tucker, Gupta, & Yeo, 2008). Top-down institutional initiatives such as these are necessary to drive transformational change. In particular, a top-down approach provides a common institutional framework for curriculum design; the provision of shared teaching infrastructure, compliance with institutional policy; implementation of standard instruments to evaluate and mange quality in teaching; ensure program sustainability and viability; and compliance with the expectations of regulatory agencies (TEQSA, 2012). Figure 4. Factors influencing transformational change in higher education teaching and learning This study has demonstrated, however, that outside factors and conflicting priorities hindered adoption of recommendations from engagement reviews, which were part of a larger strategy for transforming teaching and learning within the business faculty. This suggests the need for a feedback loop to ensure alignment of institutional initiatives and their operationalisation at the faculty level. Such a model is presented in Figure 4. It positions top-down institutional initiatives (e.g. curriculum renewal, infrastructure for teaching spaces and technology) and bottom-up operationalisation of these initiatives (e.g. faculty-based reviews, faculty-based pedagogical and technical support, scholarship of teaching and learning projects,) as complementary components of transformational change. Figure 4 also illustrates the principle that strategic institutional priorities should be well aligned with faculty-based projects. It also shows that the operational experiences of faculty-based agents of change have the capacity to inform the alignment of potentially disparate institutional initiatives and priorities. This could include, for example, recommendations to ensure new workload initiatives consider the workflows associated with online and blended delivery modes before they are rolled out; establishing institution wide recommendations for staffing and resourcing online and blended delivery modes; establishing teaching sandboxes to encourage Operational alignment of initiatives Strategic institutional priorities Institutional T&L Initiatives Transformational Change Faculty-based agents of change Top-Down Bottom-Up 630 Australasian Journal of Educational Technology, 2014, 30(6). teaching innovation without concern for the impact on end-of-semester evaluation scores; and rethinking learning active learning activities to take best advantage of technology-enhanced learning spaces before they are commissioned. Conclusions This study has reported on staff perceptions about how student engagement could be enhanced within the business faculty of an Australian university. The process was designed to build staff capacity and empower them to make informed decisions regarding new innovations that enhance student engagement. Teaching staff participating in this study were overwhelmingly of the view that the student engagement review process was likely to raise the level of student engagement, and most agreed with all or some of the recommendations arising from the review. A number of outside factors, some of which were related to other institutional priorities and initiatives, were seen to hinder the adoption of some recommendations, or to otherwise impede student engagement. These include time allocation using the university workload management system, and issues with teaching spaces and technical infrastructure. However, findings arising from the engagement reviews and this study suggest that collaborative reviews undertaken by the faculty-embedded engagement unit are effective agents of transformational change in teaching and learning. The approach works from the bottom-up in a manner that is compatible with theoretical frameworks for change and the adoption of new innovations. Moreover, this study has shown that the bottom-up approach has the capacity to build staff capacity, confidence and skills to engage students in learning, taking advantage of the affordances of technology as appropriate. Future research could be conducted in relation to the impact of this process in terms of the student experience, but this was not within the scope of this project. References Allen, I. E., & Seaman, J. (2011). Going the Distance: Online Education in the United States, 2011,. Newburyport, MA: Sloan Consortium. http://eric.ed.gov/?id=ED529948 Allen, I. E., Seaman, J., Lederman, D., & Jaschik, S. (2012). Conflicted: Faculty and online education, 2012: Babson Park, MA: Babson Survey Research Group. http://eric.ed.gov/?id=ED535214 Biggs, J., & Tang, K. (2011). Teaching for quality learning at university (4 ed.). Buckingham, UK: Open University Press. Bolt, S. (2012, October). Professional development: Then and now. Paper presented at International Association for Development of the Information Society Cogniton and Exploratory Learning in a Digital Age Conference, Madrid, Spain. Bryson, C., & Hand, L. (2007). The role of engagement in inspiring teaching and learning. Innovations in Education and Teaching International, 44(4), 349-362. doi: 10.1080/14703290701602748 Caffarella, R. S. (2002). Planning prgrams for adult learners: A practical guide for educators, trainers and staff developers. San Francisco: Jossey-Bass. Chao, T., Saj, T., & Tessier, F. (2006). Establishing a quality review for online courses. EDUCAUSE Quarterly, 29(3), 32-39. Chickering, A. W., & Ehrmann, S. C. (1996). Implementing the seven principles: Technology as a lever. AAHE Bulletin, October, 3-7. Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. AAHE Bulletin, 39(7), 3-7. Coates, H. (2007). A model of online and general campus-based student engagement. Assessment & Evaluation in Higher Education, 32(2), 121-141. doi: 10.1080/02602930600801878 Creswell, J. W. (2009). Research design: Qualitative, quantitative and mixed methods approaches. Thousand Oaks, California: Sage Publications Inc. Curtin University. (2011). Student Engagement in Blended Learning Environments Guidelines. Retrieved 19 April 2013, from http://cel.curtin.edu.au/local/downloads/learning_teaching/philosophy_teaching/student_engagement. pdf Deslauriers, L., Schelew, E., & Wieman, C. (2011). Improved learning in a large-enrollment physics class. Science, 332(6031), 862-864. doi: 10.1126/science.1201783 Donnelly, R. (2010). Harmonizing technology with interaction in blended problem-based learning. Computers & Education, 54(2), 350-350. 631 http://eric.ed.gov/?id=ED529948 http://eric.ed.gov/?id=ED535214 Australasian Journal of Educational Technology, 2014, 30(6). Dziuban, C. D., Hartman, J. L., & Moskal, P. D. (2004). Blended learning EDUCAUSE Centre for Applied Research: Research Bulletin. Retrieved from http://net.educause.edu/ir/library/pdf/erb0407.pdf Guskey, T. R. (2002). Professional development and teacher change. Teachers and Teaching: Theory and practice, 8(3/4), 381-391. Herrington, A., Herrington, J., Oliver, R., Stoney, S., & Willis, J. (2001). Quality Guidelines for Online Courses: The Development of an Instrument To Audit Online Units. Paper presented at ascilite 2001, Melbourne, Australia. 263-270. Retrieved from http://www.ascilite.org/conferences/melbourne01/pdf/papers/herringtona.pdf. Knowles, M. S., Holton III, E. F., & Swanson, R. A. (2005). The adult learner: The definitive classic in adult education and human resource development (6th ed.). Burlington, MA: Elsevier Inc. Lloyd, M. & Duncan-Howell, J. (2010). Changing the metaphor: The potential of online communities in teacher professional development. In J. O. Lindberg, & and A. D. Olofsson. (Eds.), Online learning communities and teacher professional development: Methods for improved educational delivery. New York: Information Science Reference, 60-76. Martin, R. (2012). The road ahead: eBooks, eTextbooks and publishers' electronic resources. Paper presented at ascilite 2012, Wellington, New Zealand. Retrieved from http://www.ascilite2012.org/page.php?31 Mathews, K. E., Andrews, V., & Adams, P. (2011). Social learning spaces and student engagement. Higher Education Research & Development, 30(2), 105-120. Mazur, E. (2009). Farwell, Lecture? Science, 50-51. McCarthy, S. A., & Samors, R. J. (2009). Online learning as a strategic asset. Volume 1: A resource for campus leaders. APLU Sloan National Commission on Online Learning. Retrieved from http://www.aplu.org/documents.doc?id=1877 Meyers, C., & Jones, T. B. (1993). Promoting Active Learning. Strategies for the College Classroom. San Francisco, CA: Jossey-Bass Inc., Publishers. National Survey of Student Engagement. (2012). Promoting student learning and institutional improvmeent: Lessons from NSSE at 13. Bloomington, IN: Indiana University Center for Postsecondary Research. Oblinger, D. (2005). Leading the transition from classrooms to learning spaces. EDUCAUSE Quarterly, 28(1), 14-18. Oliver, B. (2013). Graduate attributes as a focus for institution-wide curriculum renewal: Innovations and challenges. Higher Education Research & Development, 32(3), 450-463. doi: 10.1080/07294360.2012.682052 Oliver, B., Tucker, B., Gupta, R., & Yeo, S. (2008). eVALUate: An evaluation instrument for measuring students' perceptions of their engagement and learning. Assessment & Evaluation in Higher Education, 33(6), 619-630. doi: 10.1080/02602930701773034 Parr, W. C., & Smith, M. A. (1998). Developing case-based business statistics courses. The American Statistician, 52(4), 330-337. doi: 10.2307/2685436 Robinson, C. C., & Hullinger, H. (2008). New benchmarks in higher education: Student engagement in online learning. Journal of Education for Business, 84(2), 101-109. Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York: Free Press. Seaman, J. (2009). Online learning as a strategic asset. Volume 2: The paradox of faculty voices: Views and experiences with online learning, (pp.1-56). Washington, DC: Association of Public and Land- grant Universities. http://eric.ed.gov/?id=ED517311 Singh, K., Schrape, J., & Kelly, J. (2012). Emerging strategies for a sustainable approach to professional development. Paper presented at ascilite 2012, Wellington, New Zealand. Retrieved from http://www.ascilite2012.org/page.php?31 Spencer, D., Riddle, M., & Knewstubb, B. (2011). Curriculum mapping to embed graduate capabilities. Higher Education Research & Development, 31(2), 217-231. doi: 10.1080/07294360.2011.554387 TEQSA. (2012). Tertiary Education Quality and Standards Agency. Retrieved from http://www.teqsa.gov.au/ Tight, M. (2012). Researching higher education (2nd ed.). Maidenhead, England: McGraw Hill Education. Tight, M., Mok, K. H., Huisman, J., & Morphew, C. (2009). The Routledge International handbook of higher education. London: Routledge, Taylor & Francis Group. 632 http://eric.ed.gov/?id=ED517311 Australasian Journal of Educational Technology, 2014, 30(6). Tynan, B., Ryan, Y., Hinton, L., & Mills, A. L. (2012). Final report of the project e-Teaching leadership: Planning and implementing a benefits-oriented costs model for technology enhanced learning. Sydney: Australian Learning & Teaching Council. US Department of Education. (2010). Evaluation of evidence-based practices in online learning: A meta- analysis and review of online learning studies. U.S. Department of Education, Office of Planning, Evaluation, and Policy Development, Policy and Program Studies Service. Retrieved from http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf Yin, R. K. (2014). Case study research: Design and methods (5th ed.). Thousand Oaks, California: Sage Publications Inc. Zepeda, S. J. (2012). Professional development: What works (2nd ed.). New York: Eye on Education. Zepke, N., & Leach, L. (2010). Improving student engagement: Ten proposals for action. Active Learning in Higher Education, 11(3), 167-177. doi: 10.1177/1469787410379680 Corresponding author: Brian R. von Konsky, B.vonKonsky@curtin.edu.au Australasian Journal of Educational Technology © 2014. Please cite as: von Konsky, B., Martin, R., Bolt, S., Broadley, T., & Ostashewski, N. (2014). Transforming higher education and student engagement through collaborative review to inform educational design. Australasian Journal of Educational Technology, 30(6), 619-633. 633 Introduction Institutional Setting Methodology Data collection and analysis Participation Results What are staff perceptions and beliefs about online learning and student engagement? What are the key benefits and outcomes arising from student engagement review process? What factors hindered the adoption and operationalisation of review recommendations? What outside issues do staff believe adversely impact student engagement? Discussion Conclusions