Microsoft Word - [7]84-2545-4-CE-1.docx Australasian Journal of Educational Technology, 2013, 29(3). 387 ascilite Leading the quality management of online learning environments in Australian higher education Dale Holt, Stuart Palmer, Judy Munro Deakin University, Australia Ian Solomonides, Maree Gosper Macquarie University, Australia Margaret Hicks University of South Australia, Australia Michael Sankey University of Southern Queensland, Australia Garry Allan and Robert Hollenbeck RMIT University, Australia The paper presents the findings of the first year of a nationally funded Australian Learning and Teaching Council (ALTC) project on the quality management of online learning environments by and through distributed leadership. The project is being undertaken by five Australian universities with major commitments to online and distance education. Each university, however, has a distinctive location, history and profile in the sector. The first year of the project has seen the development of a quality management framework with six interrelated elements. The framework is being applied, refined and validated in the second year of the project. Allied with the development of the framework, was the conduct of focus groups at each of the five partner institutions in the middle of the first year. These focus groups composed a range of staff involved collectively in the leadership of the organisation's online learning environment. Prominence was given to the nature and value of strategic planning, due diligence conducted in selecting and mainstreaming technologies, evaluation approaches informing decision making, and the various relationships between different leadership levels and domains. A number of key issues which emerged relating to the elements identified in the framework are examined. Introduction This paper is based on the outcomes of the first two stages (i.e., year 1) of a nationally funded Australian Learning and Teaching Council (ALTC) leadership project. The project is being undertaken over 2011- 2012 and aims to design and implement a framework that uses a distributed leadership approach for the quality management of online learning environments (OLEs) in Australian higher education. The distributed leadership approach was chosen to enable the development of the framework and in turn to contribute to its implementation. The framework is, therefore, the vehicle for building leadership capacity. The project is drawing upon the combined expertise and strengths of five universities using different learning management systems and approaches to social networking and who are at various stages of deploying their next generation online learning environments. The universities involved represent different groupings of institutions in the sector and each is reliant on disparate leadership groups to successfully implement and sustain their environments. To date, the project has identified six elements which constitute a quality management framework. These elements are presented. Additionally, two leadership focus groups are being conducted in 2011. The first completed leadership focus group explored key issues relating to the elements of the framework and distributed leadership capacities, and these are examined in this paper. The second year of the project will focus on further testing, refinement and validation of the framework with specialised leadership focus groups planned around evaluation, technological innovation and strategic planning. Australasian Journal of Educational Technology, 2013, 29(3). 388 Establishing rationale and context The ALTC leadership project addresses the key question: how does an Australian university best conceive and implement, through distributed leadership structures, a quality management framework for online learning environments? In line with international developments, Australian universities have made very large investments in corporate educational technologies to support their commitments to online and distance education. Learning management systems (LMSs) have represented the focus of these institutional investments over the last decade or more. They are perhaps currently the most widely used and most expensive educational technology tool (Salinas, 2008). Many Australian universities have recently reviewed, are reviewing or will soon be reviewing their LMS technologies with a view to making decisions on the next generation of online learning environments and developments. Deliberations around the turnover of LMSs have been reported internationally (Instructional Technology Council, 2011; EDUCAUSE learning Initiative, 2011). There are two important domains which must be considered in leading the quality management of online learning environments (OLEs) in the sector, namely: the nature and value of distributed/shared leadership in educational settings which in turn presupposes a more fundamental view of what leadership actually is; and the quality management of teaching/learning in higher education, featuring the changing nature of online learning environments. Northouse (2010) provides a contemporary overarching definition of leadership and one which sits comfortably with various lines of leadership theorising: "Leadership is a process whereby an individual influences a group of individuals to achieve a common goal" (p. 3). This definition focuses attention on leadership as a process occurring between parties, that through such a process the leader exerts influence on so-called followers (who may in turn influence each other and the designated leader), that such influence occurs in group situations and is directed to achieving common mutually accepted goals. Good leadership is about positive influence, mutual respect, trust and informed agreement to the achievement of valued and clearly understood goals. It does not involve unethical or coercive pressure directed from leader to those led to accept goals of no or limited value to those most affected. A more enlightened view sees the existence of both assigned and emergent leadership. As observed by Northouse (2010), "When leadership is defined in this manner, it becomes available to everyone. It is not restricted to the formally designed leader in a group" (p. 3). There has been significant interest in new lines of leadership theorising around distributed and shared leadership. This theorising sees a paradigm shift from a focus on the leader, as individual, and his or her traits, skills, styles and behaviours in relating to parties designated as followers to a focus on the phenomenon of leadership as enacted through various parties in multiple relationships with leadership intent and capability to achieve valued goals. These parties may be those in formal leadership positions at different levels and in different functional areas in the organisation who wish and need to act in concert, or it may involve those with particular interests and capabilities emerging as informal leaders for periods of time and acting in concert with those with formal leadership authority. This new paradigmatic focus is highlighted by Pearce and Conger (2003). Leadership of this nature is seen as distributed or shared (possibly even described as dispersed or networked). While scholars in these fields may take issue, for the purposes of this paper, the terms are seen as interchangeable. Much of the interest on distributed/shared leadership has emanated from the schools sector at national and international levels (Leithwood, Mascall, & Strauss, 2009; Harris, 2009; Spillane, 2006; Spillane, 2007). As applied to the schools sector, "the core principle is one of extending or sharing leadership practice" in response to increasingly perceived structural limitations in advancing leadership and organisational performance (Harris, 2009, p. 3). Harris (2009) tracks the underlying theoretical foundations of distributed leadership to distributed cognition, social learning, network and activity theories, and outlines its scope: Distributed leadership, essentially involves both the vertical and lateral dimensions of leadership practice. Distributed leadership encompasses both formal and the informal forms of leadership practice within its framing, analysis and interpretation. It is primarily concerned with the co-performance of leadership and the reciprocal interdependencies that shape that leadership practice (p. 5). Australasian Journal of Educational Technology, 2013, 29(3). 389 Based on the scope of distributed leadership identified by Harris (2009), a number of key alignments become prominent in higher education institutions: • Vertically amongst faculty formal leaders in hierarchy; • Vertically amongst senior executive leaders and faculty formal leaders; • Horizontally amongst senior executive leaders; • Horizontally amongst faculty formal leaders across hierarchies; • Horizontally amongst senior executive leaders and across faculty leadership; • Informal academic and professional support leadership horizontally amongst staff at discipline, school, faculty and interfaculty levels/domains; and • Informal leadership at particular locations in multi-campus environments. In summing up the extensive literature on distributed leadership emanating from the schools sector, Spillane (2006) identifies three essential elements in framing a distributed perspective on leadership: Leadership practice is the central and anchoring concern; Leadership practice is generated in the interactions of leaders, followers, and their situation; each element is essential for leadership practice; The situation both defines leadership practice and is defined through leadership practice. Moreover, Hargreaves and Fink (2006) relate their commitment to sustainable educational leadership to the nature of distributed leadership through their principle that, "Sustainable leadership spreads. It sustains as well as depends on the leadership of others" (p. 95). Marshall (2006) conducted a rigorous review of leadership theorising and implications for effective teaching and learning leadership in universities for the Australian Learning and Teaching Council (ALTC). He established an explicit link between distributed leadership capacity development and teaching/learning (and by implication online teaching/learning): Efforts to develop leadership capability in learning and teaching in higher education need to…be focussed on developing the capability of all those involved in planning, implementing, or evaluating [online] learning and teaching and/or the organisational environments in which [online] learning and teaching occurs (Marshall, 2006, p. 7). Capacity building for distributed leadership is centre stage in managing the quality of OLEs: "Capacity building involves the use of strategies that increase the collective effectiveness of all levels of the system in developing and mobilizing knowledge, resources and motivation, all of which are needed to raise the bar and close the gap of student learning across the system" (Fullan, Hill, & Crevola, 2006, p. 88). The leadership of quality OLEs is becoming more complex and demanding as we see the growing size, diversity and reach of universities (some with offshore campus operations and TAFE programs), with the growing number of information and communications technologies (ICTs) which constitute such environments, with the loosening of institutional control over certain technologies which can be used for effective learning and teaching, with the greater size and more diverse composition of universities' workforces and student populations, and the ever present multiplicity of curricular and pedagogical models which underlie an ever-expanding range of occupations and professions requiring higher level education. Added to these complexities and demands is the intensifying national and international competition in the e-learning marketplace, now involving the formation of competitive alliances and partnerships with global reach. No one formal leader at the top, no matter how ambitious and knowledgeable, could possibly contend with the complexity of issues related to the quality management of OLEs. Leaders must be mobilized down, across and throughout the organisation to realise the full benefits of massive institutional investments in online learning systems. This is well expressed by O'Toole, Galbraith, and Lawler (2003): "The lesson is this: The more interdependent the work of co- leaders, the more input they should solicit from affected others and the more they need to coordinate between themselves" (p. 260), and reinforced by Conger and Pearce (2003): "Shared leadership is therefore an effective solution to a fundamental dilemma: No single individual possesses the capacity to effectively play all possible leadership roles within a group or organizational setting" (p. 285). Australasian Journal of Educational Technology, 2013, 29(3). 390 In turning to quality management in higher education, to many, the idea of applying quality concepts (particularly some of the terminology rooted in the manufacture of commercial products) to education is anathema (Anderson, 2006). For some, in the context of education, it does not seem possible to move beyond transcendent conceptions of quality. However, defining quality in education by using other equally ill-defined terms (such as excellence) doesn't advance the issue in any practical way. The contemporary, user-centred, perceived experience view of quality is not conceptually incompatible with many modern ideas in education (Lewis & Smith, 1994), including student-centred learning, inclusiveness and learner experience design. But, like all matters of educational policy and practice, the devil is in the detail, and no less so than in defining/agreeing what we mean by 'quality' in higher education, and then devising objective measures for it. Quality is a system-wide function, and a comprehensive model of quality in higher education should encompass both teaching (organisation related aspects) and learning (student-related aspects), and include input, process and output factors for both areas (Oliver, 2003). Research and development methodology The paper presents a quality management framework under development and the findings of the first leadership focus group discussions. Both are linked but strictly speaking the framework has not solely emerged from the first leadership focus group round. The project's developmental methodology needs to be outlined. Key literature, as above, was presented to project partner representatives early in the project. The project team met and presented their institutional profiles around seven factors (with a separate special issues category) which were seen to be important in managing online learning environments, and based on the literature. Key challenges and issues relevant to these factors were discussed at the meeting. Subsequently, 20 questions were identified relating to these 7 factors. A comparative analysis of institutional practices was undertaken, and the 20 questions grouped under the 7 factors were in turn related to the Australasian Council on Open Distance and E-learning benchmarking standards (Australasian Council on Open Distance and E-learning, 2010). The 20 questions were reviewed and refined to six major questions which were to inform the first leadership focus group discussions: • Does the university have a vision for learning and teaching; is there an aligned or integrated vision for technologies in learning and teaching? • Does the university have a plan or roadmap that provides the institution with a strategic direction for its OLE and, if so, how often is this plan/roadmap reviewed and updated? • What university-wide processes are in place to ensure that the introduction of new and emerging technologies is subjected to due rigor prior to being allowed on university systems? • What evaluation measures are in place to ensure the current use of OLEs are fulfilling the needs of both the staff and students? • What are the relationships between the university's vision, plans, governance structures and budgeting in relation to the OLE? • What institutional-wide mechanisms are in place for managing learning and teaching as well as technologies for learning and teaching to ensure: effective collaboration-communication between all relevant parties? Alignment with strategic directions? A focus group discussion (FGD) was held with each of the five partner institutions. Two were completely face-to-face, two were predominantly face-to-face but with a single participant linked via videoconference and for one, it was the facilitator who linked via video conference with participants (the facilitator was an external consultant with no affiliation with any of the institutions). The project representative from each partner institution chose who would attend the FGD based on an overall brief agreed to by the project team. Forty-seven of 58 (81%) invitees participated. Of these, 29 came from Central areas and 18 from the Faculties mainly through Associate-Deans Teaching and Learning or their equivalents (the generic terms 'Faculty', 'Associate-Deans Teaching & Learning' are used to cover institution variation). The most senior staff who participated were at Vice-President (VP)/Deputy Vice- Chancellor (DVC)/Pro Vice-Chancellor (PVC) level. Each Head of the relevant Teaching and Learning Centre participated and those most responsible for the technologies and governance of this area were well represented. Other attendees included university librarians and Learning and Teaching Centre staff. Australasian Journal of Educational Technology, 2013, 29(3). 391 The areas canvassed were determined by the project team (see above) with the facilitator designing the activities and approach. This was accepted by the project leaders and then affirmed after the first focus group. Oral and written data resulted from each FGD. The discussion was recorded but not transcribed. Detailed data, including the source of some comments, were provided in the individual reports that were prepared after each FGD and, where requested, these were returned to the institution for confirmation of accuracy. Each was discussed with the project leaders prior to the next FGD. Within ethical guidelines, these data would allow project team members to make their own interpretations and draw their own conclusions, as, presumably, did FGD attendees who shared views with their peers, possibly gaining further insights after the completion of the event. Framing the elements of quality management of online learning environments Various ICTs constitute an institution's OLE and therefore demand a total approach to quality management for student learning and staff teaching. Certain ICTs have been designed specifically for educational uses and are institutionally controlled and supported for the benefit of a broad range of staff and students. Other ICTs (sometimes described as emerging web 2.0 technologies) are not necessarily controlled and supported by the institution but are nonetheless used for educational purposes. Such non- corporately supported ICTs might be locally developed and supported within the institution, supported centrally by the organisation for limited selective use or are located outside the institution for open use. The total quality management of OLEs requires the broadest conception of the variety of ICTs which can be used for educational purposes and strategic approaches to the leadership of their use in sustainable and responsive ways. A quality management framework for OLEs needs to encompass the full range of domains that must be taken account of in deriving maximum value from all investments in ICTs for educational purposes. Investments cover staff and student time, production of resources, and various ICT budget expenditures on hardware, software and networks. Staff time covers all relevant academic teaching and non-academic general and professional staff throughout the organisation. There are critical questions emerging around how quality management of OLEs can be best done given the changing landscape of ICTs and the institutional demands placed on OLE leaders and leadership groups to respond to external pressures in positioning their institutions in the competitive higher education marketplace. While common components of quality management of OLEs are evident in the literature and the experiences of the partner institutions, and critical questions of shared significance identifiable, specific quality management approaches are contingent on institutional histories, current positioning and future aspirations. Thus, we concur with the following two quotes in encapsulating our view for the need for contingency-based approaches to using a quality management framework for OLEs in the Australian education sector. As Mintzberg (2004) has observed: It would be nice if we could carry reality around in our heads and use it to make decisions. Unfortunately, no head is that big. So we carry around theories, or models, instead; conceptual frameworks that simplify reality to help us understand it. (p. 249). What managers need is descriptive insight to help them choose or develop prescriptions for their own particular needs. The fact is that better description in the mind of the intelligent practitioner is the most powerful prescriptive tool we have, for no manager can be better than the conceptual frameworks he or she uses. (p. 252). With these premises in mind, Figure 1 presents the first major iteration of a quality management framework for OLEs. The framework has been principally informed by the McKinsey 7S framework (see Hayes, 2010, p. 113), and is consistent with factors identified by Bates and Sangra (2011) of importance in managing technology holistically in higher education for teaching and learning transformation. Typically, a quality framework will define the characteristics/variables of importance for a domain of interest, and identify the dimensions/values that those characteristics might take from a range of categorical states or ordinal values on a continuum. Such an approach means that it is possible to be judged as high-performing/high quality in some aspect of the framework while simultaneously being Australasian Journal of Educational Technology, 2013, 29(3). 392 judged as low-performing on others. The heart of the framework is the building of distributed leadership capacity with the aim of enhancing the quality of learning and teaching outcomes and experiences through the alignment of the six interrelated elements and the cultivation of individual and collective agency (amongst formal and informal leaders interacting in and across hierarchies). By individual and collective agency we mean that staff can take reasonable action based on their professional judgement making and in interaction with colleagues in their domains of work and sphere of responsibility. It should be noted, however, that in the context of OLEs, frameworks abound, including those specifically focussing on the quality of student online learning. They go by many names: frameworks, models, benchmarks, systems, etc (see Oliver, 2003; Marshall, 2004; Australasian Council on Open Distance and E-learning, 2010; IT Governance Institute, 2007; Marshall, 2007; Scott, 2008; Queensland Government Chief Information Office (Enterprise Architecture & Strategy), 2009; Charles Sturt University, 2010; White & Larusson, 2010). The 6 element OLE quality management framework (6EOLE QMF) is being designed to augment, indeed, help managers better implement other existing benchmarking models and standards. Figure 1. 6EOLE QMF (Quality management framework). Descriptors of the six elements follow: 1. Planning: External environmental analysis and trend spotting, strategic intelligence gathering, external benchmarking, organisational capacity analysis, institutional purpose, reputation, vision, principles, objectives and strategies, accountabilities, timelines, and resource implications. 2. Organisational structures: Nature, range, coordination and delivery of valued services (underpinned by clarity of understanding of needed expertise/staffing capabilities) for staff and students. 6EOLE%Quality%Management%Framework Resourcing Evaluation Organisational structure Planning Technologies Governance Planning Implementation Evaluation Review Improvements Building distributed leadership Quality experiences & outcomes: Aligning elements Cultivating individual & collective agency Interacting formally & informally Through & across hierarchies Australasian Journal of Educational Technology, 2013, 29(3). 393 3. Governance: Institutional, faculty and school/department committees and forums (and associated responsibilities and accountabilities), policies and standards. 4. Technologies: Type, range, integration, promotion, and innovation and mainstreaming of emerging technologies. 5. Resourcing: Maintenance and enhancement of technologies, skills recognition and staff development, media production, evaluation activities, governance mechanisms, i.e., all other elements. 6. Evaluation: Stakeholders' needs, methods, reporting, decision making through governance structures, evaluation relating to the initial selection of new technology, and evidence gathering relating to the on-going assessment of its performance, value and impact. Focus group issues The FGD covered a broad range of issues of relevance to the six elements of the framework, and several broader conclusions can be drawn before specific matters are considered. Not surprisingly, institutions are at different stages of development of their OLEs. Where they are situated on the developmental continuum is not, necessarily, a product of how long and to what extent they have been involved with online learning. Rather, it is at what stage within the leadership cycle they are at presently. Arguably, renewal of leadership is a consistent element and, when this is at a senior level, this will have significant impacts and implications for all facets of online learning. For example, major senior leadership change with accompanying new strategic directions (see Planning element) will have major impacts on all other elements in the framework through the various relationships amongst them. While elements and their relationships may be relatively stable when institutions are in a strategic steady state, all can be disrupted with major changes in organisational direction where the OLE is seen as strategically critical to success in the marketplace. Those universities that are anticipating significant senior leadership change in the next 6 months were also confident this would lead to positive outcomes and that present perceived deficiencies would be addressed. If there is acceptance that the major elements in designing and implementing a quality management framework for an OLE involve leadership ensuring a shared and understood vision, strategy and plans and the effective management of teaching and learning in that space, these discussions gave a compelling indication that at some universities, and for some key players, significant elements are missing. For one university this was expressed as a lack of vision or, as another university put it, there was no 'big picture' to guide OLEs. For another, it was a lack of an effective communication strategy and for several it was the lack of a coherent plan or roadmap. As the data below evidences, while there was a sound grasp of what should be done there was acute awareness of where the reality failed to match expectations of what was necessary. Even the assumption that a 'quality management system' existed was challenged and repudiated in some sectors. There were important differences between how institutions choose to define vision and strategy in this area. One institution, for example, has adopted a 10 year window where other institutions are presently reconceptualising such vision statements or framing them within a 2-5 year period. There are discernible tensions, also, between those who advocate that online learning strategy, policy and planning should be separate – an indication of its distinctive character and also importance – and those who argue that it should be 'integrated'/'embedded' within general teaching and learning as testimony to the fact that it is not different from normative tertiary learning. A challenge for leaders in this domain, then, seems to be to bring these views forward so they can be recognised and understood and then reconcile them within an emergent strategic view that can be sufficiently broad to accommodate different perspectives, yet sufficiently coherent and cohesive for the university's staff to have confidence that they are working within an enabling environment towards a well understood and agreed end. Fundamentally, there needs to be a shared understanding of what is required by all stakeholders so that everyone's efforts are contributing to a shared strategic direction for the OLE. It is not enough to set up strategic planning and documentation outside the users and then communicate it to them, important though that is. Distributed leadership is a model that supports having them involved throughout, and this is more likely to mean the 'why' is dealt with, including how this will enhance students' learning. Central Australasian Journal of Educational Technology, 2013, 29(3). 394 bodies need, for example, to understand more fully how their technological decisions will impact on learning and teaching; Faculty staff need to appreciate, for instance, why it is unrealistic and impractical for large scale training and trialling to be done prior to a decision of a provider of an LMS being made and that it is impossible – and undesirable if one accepts that students look for consistency most of all with their online learning – for every individual desire to be accommodated. While there is evidence of data collection, evaluation appears as a weak element for most – if not all – institutions. This accords with Bates and Sangra (2011), based on information collected from 30 tertiary institutions (including 25 universities) primarily located in North America and Western Europe, who concluded, "To our surprise, though we found that none of the case study institutions had a formal, systematic, and comprehensive strategy to evaluate its use of technology for teaching across the institution" (p. 131). For some, albeit a minority, the evaluation of the technologies, themselves, in the process leading to the decision of an LMS, for instance, has been robust but evaluation to determine the actual use to which the technology has been put and its impact on student learning is less certain. There also appears to be a significant gap between the gathering of data and good decision making and governance in enhancing the quality assurance process of OLEs and this could indicate a less mature leadership capacity. However, several institutions are apparently at the point of (re)introducing stronger evaluation regimes. This is an important leadership challenge in relation to evidence-driven change management. Turnaround leadership, Fullan and Scott (2009) argue, is dependent on the development of evidence-driven quality management systems, and a greater focus on outcomes and impact (as opposed to inputs). Thus, the nature and relationships amongst the planning, evaluation and governance elements in the framework requires close attention by institutional OLE leadership, and will be an ongoing particular focus of the project. Beyond, these broad observations from the FGDs, specific findings emerged as related to particular elements of the framework. The nature and effectiveness of strategic planning FGD on strategic planning addressed the following two matters (with results presented in Figure 2): Question (a), The importance for a university to have a plan/roadmap that provides it with a strategic direction for an OLE; and Question (b), The usefulness of their plan/roadmap (as of today and from their perspective) for their OLE. (a) (b) Figure 2. Importance and usefulness of a plan/roadmap as a percentage of numbers attending. 0% 10% 20% 30% 40% 50% 60% 70% 80% Centre Faculty 0% 10% 20% 30% 40% 50% 60% 70% 80% Centre Faculty Australasian Journal of Educational Technology, 2013, 29(3). 395 Most were confident that such a plan was essential. It provided direction, validation and assurance. Without such a strategic plan it was difficult to align all the areas and provide resource allocation (Resourcing being another key element of the framework, and strongly determined by planning and governance considerations). While different universities, with different missions and discipline areas, had different perspectives regarding the importance of online learning, in the context of universities being outward looking and global, some sort of strategic plan was essential to assist in capturing that market. From a business perspective, universities need to be more agile in terms of how they make their offers in response to the demands students have on their time and this demands a sustainable online strategy moving forward. Legal reasons, including access and equity for students, were also raised as justifying the importance of such a plan. It is worth emphasising that planning requires as much attention to business and marketing issues as purely educational ones for the institution. Those who expressed reservations did so on the basis that such planning could stifle innovation – "You can't always capture what is coming over the horizon quickly enough" - and principles may be drafted without due cognizance of technical and social innovations. Further, even though staff members could be involved in a lot of online learning, they "often travel completely unaware of what's in the clouds" and although it filtered down through work plans there was often not much/nil contact with such plans. For some, A university is a self organised system and plans/roadmaps come from a different ideology, from a control philosophy. It is absolutely vital to have some kind of word picture of where you are headed and what is required is a galvanizing vision that people can assemble themselves around. But as soon as you move into planning and road mapping I think you lose some flexibility and you also start to impose strictures on people and on organisational units. While most thought such a plan/roadmap was important, what this actually was, what it should be like, and also whether their university had such a document was less certain. As Figure 2 above indicates, there was a trend for Central staff to rate their present plan more highly than did Faculty staff, but it is significant that no one at any of the universities considered their current plan met its goals to a very good extent (5) and over half of staff (25 of the 47, with at least 3 from each institution) gave a rating of 2 and below or said there was no plan to rate. Given that discussion revealed that some staff, who had given scores of 3 and 4 had done so on the basis of plans providing operational – as distinct from strategic – direction, or that their university was on the verge of providing such a plan, that proportion is probably higher. Significantly, staff within the same institution had very different views in this area. Staff – including senior leaders with responsibility in this area – conceded there were problems and limitations with their present planning documents: "[There is a] need to ask different questions that aren't answered by our current information documents". It was claimed these documents were "not widely socialised amongst those who would actually use them" and, because "the average academic was completely unaware", these documents could not provide strategic direction. For some, the usefulness of such a roadmap was to offer strategic direction for the university; for others, its usefulness was how it was understood by, and impacted on, the staff and students using the OLE. It was agreed that alignment between users and senior management was essential. Some saw the key issue of why a particular roadmap was not useful as the failure to distribute it beyond the relevant committees to the faculties, while others saw that the problem was what was actually being distributed. The following views were expressed about the requirements for such a plan/roadmap to be considered useful. It should be: • anchored to the strategic vision and provide strategic direction; • simple and deal with a few well understood issues that can be successfully implemented; • coherent, easily accessible and based on/reflect a shared, common understanding; • articulate the 'what' and 'how' so people can find their place and be confident of their contributions; • able to reflect sustainability; • agile – able to respond quickly to emergent technologies and change; • regularly reviewed and updated so that it is a living document and responsive to change. Australasian Journal of Educational Technology, 2013, 29(3). 396 It must not: be part of a large portfolio of plans; be prescriptive or proscriptive – it must give people room to move and interpret; preclude innovation outside it. To achieve such a result requires: • strong leadership and the commitment behind that leadership with the issue elevated to the required level of importance, with it regarded as a significant priority as far as the university's planning is concerned; • organisational structure without problematic silos; • having discussions and decisions regarding technology and teaching and learning in the same spaces and having the "right" people at these "roundtables"; • equivalent investment in OLE to match the buildings, equipment, human resources invested in face-to-face learning; • a preparedness to invest in emerging technologies and innovation. These perspectives begin to establish a set of characteristics and better practices in managing the Planning element of the Framework. Applying due diligence to the selection and mainstreaming of technologies The FGDs illuminate various perspectives on managing the technology element of the framework. It was felt that early enthusiasts/innovators should not be restricted by bureaucratic procedures but, before any technology becomes system wide, it is essential that the new technology is subjected to due rigor. The processes must enhance and not stifle. Leaders need to be aware that processes to ensure optimum decision making are variously understood by different stakeholders. Those responsible for infrastructure and technical effectiveness, as could be expected, were more concerned with those elements while Faculty staff were more likely to incorporate pedagogical elements but were also aware of the importance of infrastructure. The issue of time and timing revealed that, for some, the implementation deadlines were too short to allow the sort of rigor they would really like to see. Others were sympathetic to the view that time pressure creates impetus and is a stimulus that concentrates attention and effort. In relation to the question of how 'due rigor' would be acted ideally in selecting and mainstreaming technologies, the FGD revealed the following views which begin to form the basis of better approaches and practices in managing the technology element: • That it should be led by the "right" people and involving the "right" people – however determined. • There should be the establishment of a timeframe which is realistic, staged and achievable. • An audit of the current situation/gap analysis should be undertaken to identify needs and purpose coupled with benchmarking and review of available literature. • The criteria by which any judgment would be made must be clearly articulated and after consultation with all stakeholders with appropriate consideration of pedagogical, technical, financial and cultural (including access and equity) concerns. • The product(s) should be examined and evaluated against fitness for purpose and strategic intent in the organisation using these criteria with a demonstrated organisational fit with the university's vision, strategies, plans and budget. • The need for the assessment of alternatives and the development of exit strategy. • Importance of due diligence of preferred product/supplier. • Importance of the evaluation of the system: security, scalability, sustainability, capacity, robustness, agility and its ability to meet articulated pedagogical requirements. • Need for risk analysis. • Need for consideration of support (including training). • The desirability of trials within low risk situations involving different audiences/constituencies. • The importance of evaluating impact. Australasian Journal of Educational Technology, 2013, 29(3). 397 Overall, there would be process transparency and communication: "If these things are done it is mostly invisible to us. That is a problem because we assume the worst if we don't know". FGD explored the extent to which the reality matched the ideal. Due diligence must be related to the direction the university is going and, for some institutions, the perceived lack of strategic intent in the OLE domain negated the effectiveness of the whole process (and hence the important relationship between the planning and technology elements). Rather than seeing "due rigor" as equating to effective project management: the catalyst of a project is a vision and that is what's sadly lacking and has been lost over a period of time – that uniqueness, that innovative driver (rather than the provider's contract expiring) and the leadership required for that will be in the context of what needs to be done with different types of leaders for different purposes at different times. A key issue that emerged was that Central providers looked at implementation of, for example, Blackboard or Moodle, but Faculty staff, in particular, needed to be convinced they needed Blackboard or Moodle at all. Extensive consultation was evidenced but it was recognised that, even within involved groups, sometimes people are left unaware and that people, even in the same area, could have polarised views – "IT creates religious fervour in people and people believe or they don't believe and no matter what you do you'll invariably have people who don't like what you do". It was concluded that maximum benefit for minimum risk was the goal, but that it was impossible to please everyone and, to attempt this, would be counterproductive. Further, While there can be ambitious and aspirational claims to justify such investment, and cash releasing and non-cash releasing benefits as a result, we are not very good at closing the loop and coming back and testing the deliverables against the claims. Whether such investment would actually benefit students' learning, and how would this be established, were critical issues to discern and it was generally agreed this was a significant gap at the moment. The separation between systems and the learning was stressed: high quality outcomes in terms of system management do not necessarily mean high quality learning outcomes. Central providers, while acknowledging there were some gaps and variations, were inclined to rate the processes as being undertaken to a reasonable or good (but not very good) extent. Faculty staff tended to be far more negative in their assessment with predominantly 'nil' and to 'minimal' extent ratings. Evaluating impacts for improvement FGD on matters relating to the Planning, Technology, Resourcing and Governance Elements of the Framework linked to the element relating to the evaluation of OLEs and how they enhance learning and teaching experiences and outcomes. The FGD question on evaluation was broken down into three sub- issues for the purpose of structured discussion: Question (a), The importance of evaluation; Question (b), The effectiveness of the present evaluation; Question (c), The estimated effectiveness of evaluation in around 6 months from now. The groups thought such evaluation was important, more than half (29 of 47) thinking it an imperative (very good extent). Only 6 staff from the same institution rated their present evaluation highly (good extent and very good extent), but most Central group participants thought it reasonable (reasonable extent rating), with Faculty members being less satisfied (see Figure 3 below). Australasian Journal of Educational Technology, 2013, 29(3). 398 (a) (b) (c) Figure 3. Importance and effectiveness of evaluation now and in 6 months' time as a percentage of numbers attending. It was argued, however, that no evaluation is likely to be a very good extent in practice. Waiting to do a full, thorough evaluation may mean missing the opportunity. There are tensions between being completely comprehensive and the practicalities – "you can't afford a perfect process" and "it probably can't be justified" – and, by improving things now and for the immediate future the goal posts for subsequent evaluation are moved. Again, there was not a definite shared understanding of what 'evaluation in this context' meant and entailed. It was suggested there was a conflation with a mix of organisational evaluation of functionality (including pedagogy) and governance structures. Those who considered the present evaluation ineffective did so on the basis that: • If the purpose of evaluation is continuous improvement, it needs to be culturally embedded – second- nature. Therefore, wherever you have short term reactivity and no long term purpose the effectiveness is rated at 1 and, unless the cultural perspective about quality is addressed and evaluation becomes the norm, rather than enforced, it won't be successful. 0% 10% 20% 30% 40% 50% 60% 70% Centre Faculty 0% 10% 20% 30% 40% 50% 60% 70% Centre Faculty 0% 10% 20% 30% 40% 50% 60% 70% Centre Faculty Australasian Journal of Educational Technology, 2013, 29(3). 399 • While there had been a medium successful contained testing of the technology, the University had stumbled with testing in use. "It was often conceived as an IT project when really it was a fundamental change to teaching and learning" and, despite surveys, this is not being captured sufficiently well in terms of how it is changing the experience for students and staff. • The importance level is directly related to the evaluation of impact (part of due rigour) and this is more important than the evaluation of the technology, itself. • While a university could be "very data rich with lots of evidence" this does not mean these data were used meaningfully to inform decision making. To move the rating of effectiveness upward, the following points were made: having effective leadership; achieving something tangible in a short time span; having a communication strategy that is two-way: that explains, is responsive, and engages; communicating the evaluation plan and what was going to be addressed and achieved; making staff feel really involved; prioritisation; and adequate funding. If these could be achieved, there was reasonable confidence that the situation would improve in the next 6 months (see Figure 3c) but, for several, this was too short a time frame for meaningful change to occur. These observations again provide a basis for better approaches and practices in managing the evaluation element. From above, the important link between evaluation and effective, evidence-based decision making through good governance structures is highlighted. Allied to the project, Holt, Palmer, and Dracup (2011) identified a number of conceptual, procedural, technical and organisational issues that need to be resolved in conducting an institutional evaluation of a new OLE. The evaluation element can be particularly problematic to address managerially given the various competing and vested interests in such an exercise. Situating and aligning leaders and distributed leadership Participants worked in small groups to complete a matrix covering different element and levels of leadership alignments. Given the compilation of the groups, the uncertainties and differences in key elements are noteworthy. The responses to the matrix indicated that where leadership was situated in the quality management of OLEs was variously understood and conceptualised. Within the groups, and then between the groups, it was apparent that notions of how distributed leadership of OLEs was played out varied, with some seeing a clear hierarchy of senior staff across all levels and others claiming that leadership was a role for everyone. Removing the element of individual difference, even within groups (generally of 3-4 people) in several instances the formal leadership of key elements of the matrix could not be identified - or was identified very differently - by groups from the same institution. Formal leadership of OLEs was identified in the form of position (e.g., DVC or PVC, Director or Head); area (e.g., Learning and Teaching Unit); committee (e.g., Learning and Teaching committee, ICT services); project team (e.g., Budget project team); structures (e.g., Project governance structures); documentation (e.g., Horizon 2020, Teaching and Learning Committee Policy). Informal leadership was identified very broadly across all elements from 'the academic teaching community' and 'champions of online learning and teaching' to those named by others as formal leaders (e.g., User reference groups and Associate-Deans Teaching and Learning). Associate-Deans Teaching and Learning were likely to list this group as leaders across all areas and to see their role as an important linking element. Other connections were seen in organisational structures that allowed overlap in committee membership or fostered communities of practice and through the leadership of online teaching and learning champions drawn from the Faculties as well as through effective project management. The existence of coherent and accessible underpinning documents that everyone could draw upon assisted in developing a holistic approach to OLE development, implementation and management. There was reference, however, to duplication and replication and competition for resources by the various stakeholder groups with the planning cycle being described by a senior leader at one university as "recursive and self-referential". Assessment of the extent to which the flow between the key elements was logical varied from 5 (to a very good extent) to 2 (to a moderate extent). Again there was a trend for Faculty staff to provide lower ratings than their Central staff counterparts. There was also a tendency for the extent to which the connections were well understood to be rated lower by both Central and Faculty staff than the extent to which they Australasian Journal of Educational Technology, 2013, 29(3). 400 were seen as logical. Where there was a perceived limited communications strategy, it was predictable that the relationships between the different elements would be seen differently. Similarly, where there was consensus that a particular university lacked a clear vision statement, there was a resultant lack of flow between vision and the other listed elements and an assessment of a lack of leadership. The opportunity was given for the groups to combine elements. Some universities have, for example, a clear statement of policy that online learning is fully integrated within learning and teaching in general but this was not reflected in some group responses. While there were instances where they were merged, and replication/similarity between those who kept them as separate, there were also indications of less satisfaction for the management of online learning in particular. And opportunity to change the wording of the various elements was provided (no group did this) and to add to the list. 7 additions were made: external, policy, institutional pedagogical framework, Equity and diversity, capability development, academic development, student support. Ratings of the connections between these and the level to which they were understood were provided only by those who nominated Academic development and Student support. Both received 0 ratings. The main gaps identified were in a perceived lack of: communication; a clear vision statement; fostering creative solutions and innovation; benchmarking or metrics; clarity with resentment of senior leadership; integration of service divisions; integration between OLE management and across local and overseas campuses, and in disconnects between: IT decisions and teaching and learning agendas; operational and strategic planning; vision/strategy and budgeting and between planning and budgeting. The project assumed that an OLE will be underpinned by quality management yet this was an assumption that was rejected by several. Given some form of quality management, the extent to which leadership was recognised by it, developed through it and aligned with it was, with the exception of 2 of the 13 groups and both from the same university, seen as being, at best, to a “reasonable” extent. For many, this was to a "minimal" or "nil" extent. Where groups were composed entirely of Faculty staff (not possible to achieve given the composition of some FGDs) they were harsher in their assessment. Tension between leadership from the Centre and within Faculties as well as the "over-competitiveness" between Faculties and Divisions was noted. While some groups attributed this apparent deficit to them being quite early in the process, the lack of a quality framework that is well defined and well socialised, adopted, communicated and evaluated appears to be lacking for several institutions at the moment. The 6eOLE QMF is being developed to address such a need. How such a framework once fully developed, can be best implemented is the ongoing focus of the project, along with the approaches and strategies which can contribute to building distributed leadership capacity to enable all the elements to be effectively meshed together for the maximum benefit of the learning and teaching experience. Conclusion The quality management of OLEs requires that they be strategically situated in the organisation's overall positioning in the higher education marketplace and demands that the broadest range of teaching staff and students derive value from their use. OLEs must be sustainable and responsive to changing circumstances within and external to the organisation. The planning element of the framework presented is pivotal in shaping the other elements, with critical relationships identified between planning, technology, governance, evaluation and resourcing. Focus group discussions provided views on ways of effectively managing each of the elements in the framework. Better approaches and practices were evident in the analysis. However, these need to be enacted contingently based on particular organisational circumstances. Quality management of OLEs also requires an enhanced form of distributed/shared leadership approach given the complexity of the task, and the range and types of both formal and informal leadership expertise involved in decision making. Distributed leadership capacity building must be nested within the various interrelationships amongst elements constituting the framework. Distributed leadership capacities need to be reviewed and renewed in line with major developments in OLEs. Framework elements need to be clearly understood, well aligned and developed and implemented holistically through well-articulated distributed leadership approaches. The overall responsibility for orchestrating such an approach still resides in those appointed to the most senior strategic leadership positions in the institution. Australasian Journal of Educational Technology, 2013, 29(3). 401 The quality management framework is a useful aid for those in such positions to enhance their OLEs over time. Acknowledgements The authors wish to thank their own university, Deakin University and the partner institutions (Macquarie University, RMIT University, University of South Australia, University of Southern Queensland) for their generous contribution to this project, as well as all those participants who contributed to the data collection processes. The authors also thank Dr Di Challis of Challis Consulting for conducting the focus group discussions. Support for this publication has been provided by the Australian Learning and Teaching Council Ltd, an initiative of the Australian government's Department of Education, Employment and Workplace Relations. The views expressed in this publication do not necessarily reflect the views of ALTC. References Anderson, G. (2006). Assuring quality/resisting quality assurance: Academics' responses to 'quality' in some Australian universities. Quality in Higher Education, 12(2), 161-173. Australasian Council on Open Distance and E-learning. (2010). ACODE benchmarks. Canberra: ACODE. Bates, A. W., & Sangra, A. (2011). Managing technology in higher education: Strategies for transforming teaching and learning. San Francisco: John Wiley & Sons. Charles Sturt University. (2010). CSU educational technology framework. Wagga Wagga: Charles Sturt University. Conger, J. A., & Pearce, C. L. (2003). A landscape of opportunities future research on shared leadership. In C. L Pearce & J. A. Conger (Eds.), Shared Leadership Reframing the Hows and Whys of Leadership (pp. 285-303). London: Sage. EDUCAUSE learning Initiative. (2011). 7 things you should know about LMS evaluation. Boulder, Co.: EDUCAUSE. Fullan, M., Hill, P., & Crevola, C. (2006). Breakthrough. CA: Corwin Press Fullan, M., & Scott, G. (2009). Turnaround leadership for higher education. San Francisco: Jossey-Bass. Hargreaves, A., & Fink, D. (2006). Sustainable leadership. San Francisco: Jossey-Bass. Harris, A. (Ed.). (2009). Distributed leadership: different perspectives. Dordrecht: Springer. Hayes, J. (2010). The theory and practice of change management. Basingstoke: Palgrave Macmillan. Holt, D., Palmer, S., & Dracup, M. (2011). Leading an evidence-based, multi-stakeholder approach to evaluating the implementation of a new online learning environment: an Australian institutional case study. In G. Williams, P. Statham, N. Brown, & B. Cleland, (Eds.), ascilite 2011 : Changing demands, changing directions : Proceedings of the Australian Society for Computers in Learning in Tertiary Education Conference, (pp. 602-612). IT Governance Institute. (2007). COBIT 4.1 executive summary and framework. Rolling Meadows, IL: IT Governance Institute. Instructional Technology Council (2011). 2010 Distance education survey results. Washington, D.C.: Instructional Technology Council. Australasian Journal of Educational Technology, 2013, 29(3). 402 Leithwood, K., Mascall, B., & Strauss, T. (Eds.). (2009). Distributed leadership according to the evidence. London: Routledge. Lewis, R. G., & Smith, H. D. (1994). Total quality in higher education. Delray Beach, FL: St Lucie Press. Marshall, S. (2004, December). Leading and managing the development of e-learning environments: An issue of comfort or discomfort? Proceedings of the Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference, Perth, 1-19. Marshall, S. (2006). Occasional Paper. Issues in the development of leadership for learning and teaching in higher education. Carrick Institute for Learning and Teaching in Higher Education. Retrieved from http://www.olt.gov.au/system/files/grants_leadership_occasionalpaper_stephenmarshall_nov06.pdf Marshall, S. (2007). E-Learning maturity model - Process descriptions. Wellington: Victoria University of Wellington/New Zealand Ministry of Education. Mintzberg. H. (2004). Managers not MBAs: A hard look at the soft practice of managing and management development. San Francisco: Berrett-Koehler. Northouse, P. G. (2010). Leadership theory and practice. London: Sage. Oliver, R. G. (2003). Exploring benchmarks and standards for assuring quality online teaching and learning in higher education. Proceedings of Open and Distance Learning Association of Australia Biennial Forum. (pp. 79 - 90). Canberra, ACT. Open and Distance Learning Association of Australia, Incorporated. Retrieved from http://ro.ecu.edu.au/cgi/viewcontent.cgi?article=4278&context=ecuworks O'Toole, J., Galbraith, J., & Lawler, E.E. (2003). The promise and pitfalls of shared leadership when two (or more) heads are better than one. In C. L Pearce & J. A. Conger (Eds.), Shared Leadership Reframing the Hows and Whys of Leadership (pp. 250-67). London: Sage. Pearce, C. J., & Conger, J. A. (Eds.). (2003). Shared leadership reframing the hows and whys of leadership. London: Sage. Queensland Government Chief Information Office (Enterprise Architecture & Strategy). (2009). Asset lifecycle - Draft 0.0.4. Brisbane: Queensland Government Chief Information Office. Salinas, M. F. (2008). From Dewey to Gates: A model to integrate psychoeducational principles in the selection and use of instructional technology. Computers & Education, 50(3), 652-660. Scott, G. (2008). Review of Australian higher education request for research and analysis -University student engagement and satisfaction with learning and teaching. Penrith: University of Western Sydney. Spillane, J. P. (2006). Distributed leadership. San Francisco: Jossey-Bass. Spillane, J. P. (2007). Distributed leadership in practice. Teachers College Record. White, B., & Larusson, J. A. (2010). Strategic directives for learning management system planning. Boulder: EDUCAUSE Centre for Applied Research. Corresponding author: Dale Holt, dale.holt@deakin.edu.au Australasian Journal of Educational Technology © 2013. Please cite as: Holt, D., Palmer, S., Munro, J., Solomonides, I., Gosper, M., Hicks, M., Sankey, M., Allan, G., & Hollenbeck, R. (2013). Leading the quality management of online learning environments in Australian higher education. Australasian Journal of Educational Technology, 29(3), 387-402.