Microsoft Word - 2196-9247-1-LE Australasian Journal of Educational Technology, 2016, 32(2). 15 ascilite Fold-back: Using emerging technologies to move from quality assurance to quality enhancement Simon N. Leonard, Robert N. Fitzgerald, and Matt Bacon University of Canberra, Australia Emerging technologies offer an opportunity for the development, at the institutional level, of quality processes with greater capacity to enhance learning in higher education than available through current quality processes. These systems offer the potential to extend use of learning analytics in institutional-level quality processes in addition to the widespread focus on business analytics, and to deliver well-constructed mixes of information from different data sources. Borrowed from music amplification, the term fold-back is proposed as a way to describe such a mix. This paper begins the design-research project of designing effective fold-back systems by expanding the theoretical assumptions about learning embedded in higher education quality processes. A number of theories building on Vygotsky’s cultural- historical approach are discussed to imagine quality in higher education in terms of what students actually do and how they engage in addition to what the institution does. The discussion is summarised in a fold-back matrix capturing the sorts of evaluation questions the systems might address. The paper concludes by providing two initial design sketches for re-purposing emerging technologies with the capacity to support expanded quality processes in education. These sketches are based on the Experience Application Programming Interface (xAPI) and Dedoose technologies. Introduction This paper reports on the early stages of a design-research project investigating the use of emerging technologies to improve quality processes within our own university. Design-research, or design-based research (DBR), is an approach that seeks to increase the impact and transferability of educational research (Anderson & Shattuck, 2012; Barab & Squire, 2004; Sandoval & Bell, 2004). It stresses the need for theory building alongside the development of design principles that direct and improve both practice and research. Consistent with the design-research approach, our intention is to share aspects of the project that may have saliency in other higher education settings. This continues a growing interest in the interplay of design and evaluation in various aspects of educational technology in recent years (Burrows & Shortis, 2011; Cheung & Hew, 2011; Kay & Knaack, 2008; Phillips, Kennedy, & McNaught, 2012). The Australian Government funded this initial stage of the project as part of its structural adjustment support to assist universities in responding to policies of mass higher education and greater competition between higher education organisations (HEO). These polices have foregrounded quality as a matter of concern in higher education (Pitman, 2014), although what is meant by quality has been a matter of significant divergence ranging from concern about academic standards due to the increased size of the sector, to an increased concern for meeting the student-customer needs in an increasingly competitive sector (Elassy, 2015). It has been argued that this vagueness around the meaning of quality, combined with a general consensus that it is important, has seen many stakeholders in higher education shift the focus of the quality discussion away from the substance of the matter (what is quality), and towards a narrower interest on how to measure it (Saarinen, 2010). The result of this technical focus has been a tendency to import a quality assurance model somewhat uncritically from the world of business, and to focus on the service delivery of the HEO and its staff through relatively blunt proxies for quality such as student retention, student success rate, and student satisfaction with course experience (Coates, 2005; Shum & Ferguson, 2011). In beginning our design work, we found that our own institution’s quality processes were consistent with this critique found in the literature – we had institutional systems, largely mandated by or responsive to government policy, that provided blunt measures of university facilities, systems and staff, but that that told us very little about the quality of learning and what the students were actually doing as part of their learning engagement. In pointing this out, we are not discounting the assessment carried out within courses, or the scholarship on teaching and learning carried out by numerous staff. At the institutional level, however, it is evident that our quality processes reify (Sandoval, 2004) a very particular instructivist concept of education. This understanding of Australasian Journal of Educational Technology, 2016, 32(2). 16 ascilite education assumes that high quality teaching leads to high quality learning, and that the responsibility for quality in the learning environment lies primarily with the teaching staff. In contrast to the instructivist understanding, however, the work of our design team is informed by the constructivist understandings of learning emanating from contemporary research in education, psychology and neuroscience that position student activity, affect and context as a significant part of the phenomena of learning (Coates, 2005; Engeström, 2006; Hutchins, 2010). As a result, the first task for our design-research project became designing the problem (Ablin, 2008) of accounting for quality in the institution in a way that includes an expanded, and possibly an expansive, understanding of learning. Designing the problem is a common part of design work in many fields and involves designers working with stakeholders to develop new ways of understanding what needs to be achieved through the design work. In this case we arrives at the need for an evaluative focus that is in harmony with current knowledge about human learning from psychology and neuroscience; that can sustain a diversity of effective pedagogical practices; that is relevant in diverse cultural settings; and that is responsive to changing funding arrangements and internationalisation (Coates, 2005). A way to emphasise the deliberate reshaping of the design problem, to flag to all involved the need to shift assumptions, was to adopt the phrase quality enhancement (QE) rather than quality assurance (QA) (Elassy, 2015; Wong, 2012). This led to following research questions to explore beside the design: 1. How can contemporary understandings of human learning be used to expand HEO quality processes in ways that enhance, as well as assure, quality? and 2. In what ways can quality systems add value to the learner, the learning community, and the learner’s communities? These questions were investigated through a design jam process (Light & Akama, 2012; Strickdorn & Schneider, 2010) in which the design team worked collaboratively and intensively to prototype and appraise workable frameworks. Presenting the final framework and its rationale form the major part purpose of this paper. The design jam process also led to our use of the term fold-back to describe the purpose of an expanded institutional quality process. Fold-back is adapted from its use in music amplification systems where fold-back speakers are used to allow performers to hear the various layers of the performance, including their own contribution, over the potential noise of the loudest components. As we use it, fold- back highlights the limitations of focussing on just one layer of information within an educational environment, when multiple layers of information are available and could be used to expand our understanding of the constructivist experience. The fold-back concept will be used to as a conceptual tool to assist the transition from an expanded quality framework to its use within institutional quality processes. To make the first steps in testing this framework, this paper also reports on initial design work using the fold-back framework to assist designing the use of new technologies in institutional quality processes. The nascent field of learning analytics is already demonstrating that new technologies can greatly assist in small and large scale evaluation within educational environments (Siemens, 2014). Our purpose here, ahead of further empirical work at scale, is to illustrate how emerging technologies such as the Experience Application Programming Interface (also known as the Experience API or xAPI) (Advanced Distributed Learning, 2014) and the mixed methods software solution Dedoose (Dedoose, 2014) can be used to support fold-back in institutional quality processes. Although designed for other purposes, they have the capacity to provide direct, immediate and granular information that can better assist in decision making about learning and teaching. Expanding the quality framework for quality enhancement The purpose of the first part of this paper is to design the problem of accounting for quality in higher education in a ways that enhance quality by taking greater note of contemporary understandings of human learning and the different types of values created by the educative process. To begin this argument for an expanded approach to quality, we begin by examining some of the limitations of current approaches. Our purpose is not to provide an extensive literature review on or history of current quality processes as this can be found elsewhere (e.g., Pitman, 2014). Rather, our purpose is illustrative and to provide a context for the design-research work that is the main focus of this paper. Australasian Journal of Educational Technology, 2016, 32(2). 17 ascilite Once a controversial topic, quality has become an everyday part of the higher education landscape (Saarinen, 2010). In Australia, the universal adoption of quality processes has been formalised in response to the government’s Tertiary Education Quality Standards Agency (TEQSA) and it predecessor the Australian Universities Quality Agency (AQUA) and a number of government mandated accountability measures that pre-date these agencies such as the Course Experience Questionnaire (CEQ) (Ramsden, 1991), a survey of graduates. Formal quality processes in most institutions are shaped by the requirement to respond to the audits of such agencies which focus almost exclusively on the activity of the institutions and their staff, and pay little attention to what students actually do (Coates, 2005). This approach means that while we may know about the resources a HEO can provide, current processes may not tell us if students regularly encounter and engage in authentic learning activities (Kuh, 2003). Even student level and student generated data in use is typically blunt and not always useful for informing decisions about teaching and learning practice. Student destination data, for example, provides useful information in evaluating how fit-for- purpose higher education organisations and courses are for preparing graduates for work. The usefulness of the data in informing decisions within courses, however, is diminished both by the time lag, and by the many confounding variables such as institutional reputation and variations in labour market demand. Measures such student retention and student success may also be difficult to interpret at the course level – high student success for example, may simply indicate a lowering of academic standards. For more than 20 years now, the major source of student-generated data for quality processes in Australian higher education has been the CEQ (Ramsden, 1991). The questionnaire is an example of similar student evaluation of teaching surveys (SETs) used around the world that are reported to provide diagnostic feedback and other information about teaching and learning experiences (Bennett & Nair, 2010). This CEQ however, also highlights the embedded conjecture (Sandoval, 2004) about learning which dominates the current quality processes. The instrument includes questions regarding the activity of teaching staff such as “The staff put a lot of time into commenting on my work.” and “My lecturers were extremely good at explaining things.” and also questions about perceptions of generic course outcomes such as “The course improved my skills in written communication.” and “The course sharpened my analytical skills.” Embedded in this approach is the assumption that certain forms of teaching behaviour will lead to high quality learning (Coates, 2005). What cannot be derived from this form of survey is any information about the role of student activity in quality learning. There are no questions, for example, about the students’ cognitive, affective or social engagement in learning activities, or their actual use of support facilities and services. The theory of learning here is one of tabula rasa in which the student is positioned as a somewhat passive receiver of the activities of teaching staff, rather than an active participant in quality learning. Given the importance placed on student activity in contemporary theories of learning (Hutchins, 2010), it is not clear that the tightly focussed feedback of the CEQ is adequate to support not simply quality assurance, but also quality enhancement (Filippakou & Tapper, 2008). Fold-back versus single channel feedback To highlight the limitations of current approaches, we have adopted fold-back as a metaphoric substitute for feedback in our design work. As we have noted, our use of the term is borrowed from its use in amplification. In that context, feedback is the squealing noise heard when a microphone is placed too close to the speaker, it is a punishing sound on the human ear and can ruin a performance. Fold-back, on the other hand, is provided through speakers that are placed to avoid the squeals of feedback while allowing the various performers to hear an appropriate mix of the overall sound, including highlighting their own performance, above the noise of the loudest instruments on stage. We contend that educational institutions need this multi-channel fold-back rather than feedback. In the educational context, feedback is used to describe the meta-language of learning (Hayes, Lingard, Mills, & Christie, 2006). This is typically dominated by feedback to students (from teachers and perhaps parents) and feedback to teachers (from students, parents and supervisors) but may involve other directions of communication. Feedback in all its forms can have powerful effects on a learning environment, although the effects can be both positive and negative (Hattie & Timperley, 2007). In this context the idea of fold- back provides an excellent description of what we are really trying to design in this space: a system which provides an appropriate mix of information that allows all of the players to hear each other, and themselves, over the noise within the environment, and while avoiding the punishing sounds of closed-loop feedback. In short, we are using fold-back to describe a system that provides information about the various layers of the educational environment to all of the different actors within that environment. In this context, the term Australasian Journal of Educational Technology, 2016, 32(2). 18 ascilite feedback remains an entirely appropriate description of individual specific channels of meta-language communication. As we return to a discussion of the CEQ, however, we will argue that just as in the audio world, feedback at the institutional level can actually ruin the performance of learners by valorising certain types of learning activity over alternative activities. The research on the use of SETS surveys such as the CEQ is now extensive and much has been learned about their use. It has been found for example, that surveys taken in class yield a higher satisfaction rating than those taken online (Dommeyer, Baum, Hanna, & Chapman, 2004; Nowell, Gale, & Handley, 2010) and that response rates to the surveys can be improved by effective communication strategies at the institution or faculty/college level (Bennett & Nair, 2010). Numerous studies indicate that this communication must indicate the ways in which a course has been improved based on previous feedback (Nair, Mertova, & Pawley, 2010; Symons, 2006). The research effort has revealed indicators of high and low satisfaction (Kane, Williams, & Cappuccini-Ansfield, 2008) and has led to the development of strategies to improve student satisfaction (Leckey & Neill, 2001; Nelson, Smith, & Clarke, 2012; Scott, Shah, Grebennikov, & Singh, 2008). Notably prominent in the factors that improve satisfaction are: the presence of a supportive peer group; a consistently accessible and responsive staff; clear management of student expectations; and just-in-time transition support (Scott et al., 2008). SETS appear highly responsive to strategies that support students to complete university assessment at their own convenience and that provide a high degree of scaffolding. They appear less responsive however, to strategies consistent with higher-order learning such as increasing the challenge to search for unrevealed information, or requiring students to utilise personal strengths for the benefit of the team (Crockett, Jukes, & Churches, 2011), and such skills are not present in the literature on how to improve student satisfaction as reported through surveys. With student surveys such as the CEQ dominating current quality processes, there is a structural incentive for HEO’s to focus on highly scaffolded learning, but little incentive to develop higher-order skills. If not managed properly, the risks of an increasing focus on the activity of teaching staff may create the perverse situation where the system that initially supports learning (Hattie & Yates, 2014) ultimately hinders it by encouraging teaching staff to continue to base their activities on the sorts of behaviours favoured in CEQ type surveys rather than activities consistent with contemporary learning theory. This risk is evident in studies that are able to correct for selection biases by randomly assigning students and tracking them through subsequent courses, there is a negative correlation between SETS evaluation and student performance in later years of a course (Carrell & West, 2010). The strong desire of teaching staff to perform (Marshall, 1999) well in such measures is highlighted in the policy literature. In this era of a broader audit culture of quality assurance (Power, 2009), both public and policy discussion of learning has become highly focussed on improving educational outputs. This adoption of the language of economics (Allais, 2011) through global policy discourse (Lingard, 2011) has led to a widespread tendency to choose proxies for educational performance in the belief that observing these proxies will lead to more effective and more efficient delivery of education. The proxies include measures such as standardised test scores in the school sector, and student satisfaction rates in higher education. This approach to educational improvement, however, has significant limitations. Sociological analysis, for example, tells us that when participants within a competitive system know the proxies in use then the proxies naturally become the objective. This limitation is highlighted in Campbell’s Law in which he argued: The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor. (Campbell, 1976; p. 49) This law has been demonstrated in a variety of educational contexts. Berliner (2009) for example, has demonstrated that high-stakes testing is leading to a narrowing of the curriculum in schools in the United States, while Lingard and Sellar (2013) have detailed the way in which some Australian states have gamed their relationship with the federal government to receive reward funding and protect their reputational capital over and above achieving real improvements in children’s learning. Similarly, in higher education, staff are being set performance measures derived from the quality measures rather than from learning theory (Miller & Seldin, 2014), with research showing that even statistically insignificant variation in those measures have an impact on academic careers (Boysen, Kelly, Raesly, & Casner, 2014). In our own Australasian Journal of Educational Technology, 2016, 32(2). 19 ascilite institution, for example, the major performance expectation for academic staff with regards to teaching is that 80% of students who voluntarily respond to an online SETS survey based on the CEQ will agree that our teaching was effective. Prizes are even awarded to staff that achieve the highest proportion of student satisfaction. In contrast there is no requirement to, or advantage in, demonstrating how participation in our courses has contributed to the development of deep understandings or skills that can be transferred to later study or life beyond graduation. The setting of targets based on proxy measures in this way is a QA method drawn from business rather than education, and while there is evidence that the approach is improving systems and processes in Australian higher education, the use of quality audits of this nature are not necessarily improving student experience or learning (Shah, 2012). To increase the capacity of quality processes to enhance learning in higher education settings, there is a need to avoid an over-reliance on any one data source, or one theory of learning, and to develop approaches for using multiple data sources that respond to the complexity of learning environments. In terms of our fold-back concept, single source feedback can become a loud and distracting squeal that dominates the performance, where what our performers actually need is a well-balanced flow of information that is folded back to them. An instructive example of the fold-back we imagined in the design jam was found in a study by Ballantyne, Borthwick, and Packer (2000) who describe a project to develop booklets that were jointly authored by teaching staff and students to set out the differing priorities for the course. The study reports an increase in the efficacy of the meta-information that flowed to teachers following the adoption of this process. Such an approach acknowledges the students’ affective response to high levels of support, but also provides a means to communicate how that support needs to be progressively withdrawn for the benefit of the students’ learning. The approach also acknowledges that students are part of the learning environment and not just a passive recipient of an educational product. In the next section we will develop a framework for the development of fold-back approaches that expand the potential of quality processes to enhance quality. Expansive and active theories of learning While the fold-back metaphor is useful, the primary task in this phase of the design-research project is to develop a framework that expands the potential of quality processes, and can guide further design work on how new technologies can support his work. The framework we developed through the design jam process is in the form of a matrix that draws upon a number of theoretical positions to present a range of suitable evaluative questions that can guide an expanded quality process. The matrix is presented in Table 1. The questions presented in it provide an alternative, although complementary, basis to evaluating the quality of learning conceived of not just as the performance (Usher, 2006) of narrow proxies of learning, but rather as an expansive activity in the sense used in socio–historical activity theory (AT) (Engestrom, 2001; Murphy & Rodriguez-Manzanares, 2008). AT, the first theory informing the matrix, builds on Vygotsky’s (1978) theory that the human mind is constructed through interactions with the world. From this perspective, people are a part of their socio-historical context and their behaviour cannot be understood separately from it. They are not just surrounded by a context, they interact with and change that context, and the context changes them. This interaction, this activity, occurs through the use of tools. Tools can be physical tools and used to manipulate physical objects, or they can be psychological tools such as concepts or plans used to influence both other people and ourselves (Verenikina, 2010). Understood as an expansive activity, learning is about acquiring the tools of the learner’s socio-cultural context and “growing into the intellectual life of those around them” (Vygotsky, 1978; p. 88). The use of those tools though, only makes sense when used for activity that is the interaction between the learner and their context. In this formation, learning is not an isolated product or performance, but rather is the integration of concept, learner, and their community. Learning connects upwards, downwards and sideways (Engeström, 2006). From this standpoint, accounts of the quality of educational programs will not simply to measure the program’s ability to develop skills in the use of particular tools, but to go further and provide an account of how the learning experience is expanding the range and improving the value of a learner’s activity. Lave and Wenger’s concepts of situated cognition and the community of practice (Lave & Wenger, 1991; Wenger, 1998) build on a similar conceptual base. This approach views learning as engaging in problem solving in the course of ongoing everyday activities. It emphasises the need for those who would foster learning to cultivate both community and networks (Wenger, Trayner, & Laat, 2011). In this model the Australasian Journal of Educational Technology, 2016, 32(2). 20 ascilite work of community is to develop learning partnerships and agendas that specify why people are involved, and what can be achieved by learning together. The work of networks, on the other hand, is to optimise the connectivity between people and so improve access to information flows and connections. This is more than a vague feeling that working with others is a good thing, it is a development of the concept that the tools acquired through learning are only meaningful when applied in activity, recognising that all human activity involves others in some way. A somewhat more controversial implication to be drawn from this work is that, the way we think is dependent on the ways those around us think. The importance of using learning in collaboration with others as a part of the value of education is increasingly being recognised. In their work on professional capital, Hargreaves and Fullan (2012) for example, have provided strong evidence that improving social capital, such as the ability to solve problems in collaboration with others, is actually a more effective way to improve the practice of teachers than improving their knowledge and skills. Such findings are not limited to school teaching and collaboration is increasingly understood as an essential skill in the twenty-first century workforce (Crockett et al., 2011). From this perspective, quality processes should provide an account of how community and networks are created and developed, and also an evaluation of the value that those communities and networks produce. Measures that overly reinforce individual performance may be counter to this need. The accounts suggested by activity theory and the community and network approach of Wenger and colleagues (2011) require evidence in the form of observable change. High quality learning will be indicated by evidence that learners can use new tools, can use existing tools with greater efficiency or efficacy, and can make use of their communities and networks to solve problems in their real-life activity. The common thread of change is consistent with the view of learning in variation theory (VT). VT has developed from the epistemology of phenomenographic research and shares the same assumptions about the nature of learning. In this approach, learning is seen as the expansion of awareness in that learners become aware of additional aspects of a phenomena, a concept or a skill, that they had not previously discerned (Marton, 1986). Variation in the understandings of concepts or skills is seen as related rather than independent with all understandings taken to be a fragment of the whole human perception of the phenomena (Akerlind, 2005). Variation emerges due to the way differences in experience and context influence the aspects of a phenomena that an individual can discern. Given this, variation theory positions understanding as context-sensitive and seeks to develop understandings from less complex to more complex - position that is also taken by the SOLO taxonomy (Newton & Martin, 2013). Variation theory argues that the most effective way to help students understand a concept is to focus on providing opportunities for students to experience variation in the features of the concept that they currently take for granted (Marton & Tsui, 2004). Flowing from this is the idea of the threshold concept or the level of discernment at which the “procedures of the subject makes sense … when before they seemed alien.” (Davies, 2006; p. 77) These expansive and active psychological theories of learning are supported by the more recent developments in neuroscience such as enacted or ecological theories of cognition (Hutchins, 2005, 2010). This work contends that the cognitive processes involved in doing work, becoming expert, and in evolving work practices are in fact the same cognitive processes. Enacted cognition is located not in the human mind alone, but in the mind-body-world systems with which we interact. With this emphasis on mind-body- world systems, this work also points to a need for quality processes that are context-sensitive and that can capture the complex nature of active learning. Australasian Journal of Educational Technology, 2016, 32(2). 21 ascilite Applying the theoretical base to a design framework The range of theoretical work summarised in the previous section guided our response to the design – research question of how contemporary understandings of human learning can be used to expand HEO quality processes in ways that enhance quality. Together these theories of learning call for quality processes that account for:  how learning experiences are expanding and improving learners’ activity,  how the HEO is creating and developing networks and communities,  the cognitive, intellectual and social contexts or mind-body world systems of learning, and  the ways in which students reach disciplinary threshold concepts. To address these needs, our design team has drawn together three bodies of work that each address some of these needs into the matrix presented in Table 1. Specifically we have made use of the conceptual framework for assessing value creation in networks and communities developed by Wenger et al. (2011), the understanding by design (UbD) approach of Wiggins and McTighe (2011), and the ideas around 21st century fluencies, developed by Crockett et al. (2011). The major structure of our matrix is borrowed from the work on value creation of Wenger et al. (2011). As understood in the theoretical understandings of learning discussed in this paper, learning is a complex social activity in which action, interaction, and context matters. It is often long-term and the result of many activities. Given this, it is difficult to demonstrate a cause and effect relationship between learning and the resulting value in later study, life, and work. In the absence of clear cause and effect mechanisms, and even of clear social agreement on the desired effects education should have, quality processes must provide a more complex account of what is happening within HEOs, and what types of value are being developed. The work of Wenger and colleagues (2011) provide a useful basis for doing this. Their framework suggests five cycles of value creation: the immediate value of activities and interactions; the potential value of the knowledge capital that is produced from the activities and interactions; the applied value that is seen when knowledge capital is put to use; the realised value that is seen in improved performance in real work and life; and the reframing value when new skills and understandings allow the reframing of success. In our design jam we saw this approach as highly complementary to the idea of fold-back as shown in Figure 1. The figure suggests that the various information and data gathered on each layer or cycle of learning can be folded back to provide an ongoing evaluation of the quality and value within an educational environment. Figure 1. Fold-back and value creation following Wenger et al. (2011) By itself this value-creation framework provides an excellent approach to capturing the value of networks and communities, which is it main intent. It also provides a strong basis for accounting for the activity of learners and the contexts of learning. A difficulty with largely open-ended accounts as proposed by Wenger et al., however, is their tendency to become idiosyncratic and not allow for comparisons between institutions. From the philosophical stance of many educators, idiosyncratic evaluations are desirable, but it is a reality of contemporary education policy that evaluation systems must allow comparison. Comparison is effectively part of the design brief. Our matrix in Table 1 addresses this by adding further focus through the combining the value creation framework with the UbD and 21st century fluency approaches. Australasian Journal of Educational Technology, 2016, 32(2). 22 ascilite Table 1 A fold-back matrix Experiences Understandings Transfer Skills Immediate Value - What happened? Establishing the learning group or community creates the first cycle of value. Activities in this phase are often most evaluated, but it should be noted that this is a phase that precedes the production of tools. The phase typically establishes the opportunities for learning. In its initial activities how, and to what extent, does the program:  Encourage participation and engagement?  Make the activity fun or relevant?  Begin to foster community and network?  Establish what can be learned together? In its initial activities how, and to what extent, does the program:  Explore and engage prior experience and understandings?  Connect to the context or the future and create significance?  Provide the conceptual building blocks to allow engagement in complex tasks?  Begin a chain of threshold concepts? In its initial activities how, and to what extent, does the program:  Require authentic interaction with team members, real or virtual?  Establish the need to search for information?  Lead to real-world problems, be solutions focused?  Create a need for original or creative solutions?  Allow students to make (critical) use of diverse media? Potential Value - What has the activity produced? At some point in an education program students begin to use the new tools they are developing to produce new products or plans or to begin to solve problems, for example, an argument in an essay, the solution to an engineering problem or the initial sketch for a new art work. As students produce new products and plans, how and to what extent do they:  Demonstrate the use of new tools and methods (Is there evidence of a change in skills, knowledge, dispositions)?  Demonstrate improved confidence in ability to work in the domain?  Use the access to new people and ideas rather than relying completely on old networks and concepts? Within the production and materials produced:  What is the evidence of changes in planning?  What is the evidence of students’ understanding of threshold concepts?  Are new approaches to problems developed?  What gaps are evident in students’ understanding? In initial production and/or planning, do students show an awareness of:  Multiple sources of information?  Their own attributes and how to use them to benefit the team?  The bigger problem beyond the immediate task?  Diverse media available for communication?  The use of imagination to discover original ideas or concepts to produce new products or processes? Applied Value – Can students put new knowledge/skills to use? Applying knowledge/skills in different situations can lead to changes in actions, practices, tools, or organisational systems. This may be referred to as authentic learning, and successful application will always reflect a deep understanding of the concepts being learned and the connections between them. Accounts of the quality of this cycle of value may consider opportunities to apply knowledge/skills within a course, or may consider evidence of student activity after a course. Where do students/ graduates:  Apply skills acquired?  Use products of the community/network they have connected with, or make tools available to their community/network/ organisation?  Draw upon their community/network to support their cause or project? When applying new skills/knowledge in diverse contexts:  What is the most significant change? Does it suggest a deepening of understanding?  Is there evidence of use of multiple concepts in relationship when solving problems?  What is the evidence of change in both the student and the context?  Does a lack of change indicate gaps in understanding or the absence of threshold concepts? Does the application require:  The incorporation of information from multiple sources?  The use of personal attributes for the benefit of the team?  The discovery of unrevealed knowledge?  The creation of a product that is part of a problem-solving process?  The use of a variety of media?  The justification of strategies and media used?  The use of imagination to discover original ideas or concepts to produce new products or processes?  Revisiting, reworking and revising to improve the solution? Australasian Journal of Educational Technology, 2016, 32(2). 23 ascilite Realised Value – Does performance improve? The capacity to apply new ideas to practice does not guarantee improve performance. Quality processes should account for the effects the application of new knowledge/skills is having on what matters to stakeholders. Characteristic of this phase is the capacity to respond to new issues or contexts and to step beyond the initial constraints and scaffolds of the learning context. This phase shows a consolidation and deepening of knowledge and skill. Where or when do students/ graduates:  Achieve something new?  Improve on some metric used to judge their performance?  Assist their organisations in achieving something new? When something new is achieved, or something is improved:  What is the evidence of change in both the student and the context?  What knowledge/skills or connections have been used to create the change?  Is there evidence of abstraction and use of learned concepts in new, novel and untaught ways? In authentic contexts, is their demonstrable improvement by either individuals or groups in:  The incorporation of information from multiple sources?  The use of personal attributes for the benefit of the team?  The discovery of unrevealed knowledge?  The creation of a product that is part of a problem-solving process?  The use of a variety of media?  The justification of strategies and media used?  The use of imagination to discover original ideas or concepts to produce new products or processes?  Revisiting, reworking and revising to improve the solution? Reframing Value – Do stakeholder redefine success? Wenger et al. (2011) suggest that a further cycle of value creation can be found in the capacity of individuals, communities and organisations to reframe their goals and to redefine success. Accounting for such change appears ambitious, but quality processes should look to account for such changes. Where or when do students/ graduates:  Reflect on what matters?  Translate new understandings into organisational or institutional changes? In reframing and redefining:  What is the evidence of changed understandings in both the student and the context?  What knowledge/skills or connections have been used to create the change?  Are changes in understanding significant? Is there evidence of changes in worldview, or a fresh understanding of the domain? Is it big? What new skills, knowledge, interests and dispositions have been created within the learners’ environments? The UbD approach of Wiggins and McTighe (2011) is a well developed practical application of the concept of learning as change. This approach draws from cognitive rather than socio-historical psychology but is never-the-less complementary. It proposes that learning should be aimed at developing deep understandings of the big ideas and core concepts, and also skills that can be transferred and used for new problems in new contexts. The cognitive science on which this approach is built notes that experts tend to understand problems first in terms of big ideas whereas novices are more likely to seek a pre-defined formula that will fit. Novices, however, cannot jump to thinking like experts without first mastering the key concepts that make up the big idea (Hattie & Yates, 2014) and so education programs should scaffold the development of the major understandings within an area of knowledge or a discipline. Ultimately though, the assessment of student learning should be in un-scaffolded performance requiring students to apply their learning in unfamiliar settings (Wiggins & McTighe, 2011). The key UbD concepts of understandings and transfer skills form the columns in the matrix in Table 1. This encourages a focus in quality systems built on this design framework to provide an account of how the activity of learners connects to cognitive, intellectual and social contexts, and also how they address threshold concepts. Finally we have used the work on 21st century fluencies of Crockett et al. (2011) to shape the dot points within the matrix. As with UbD, this work is a well-developed practical synthesis of contemporary learning theory and maintains a vision of learning as connected to the world. Its use in the matrix provides a way to Australasian Journal of Educational Technology, 2016, 32(2). 24 ascilite focus attention on how learning experiences are expanding and improving learners’ activity, and to avoid completely idiosyncratic accounts while still allowing accounts that are responsive to their context. Two initial prototypes Our purpose in this article has been to present the first stage of a design-research project seeking to use emerging technologies to support improved quality systems within our own university and other higher education organisation. To do this we have used the design jam process that has led to a focus on quality enhancement rather than simply quality assurance, an organising idea of fold-back, and the creation of a matrix of evaluative questions that can guide the quality processes designed. The matrix provides a set of questions that should connect quality processes to the contemporary expansive and active theories of learning set out earlier in this paper. The object has been to go beyond simply looking at how to improve current practice, with its embedded instructivist assumptions, but to expand current quality processes in ways that take into account current knowledge about human learning from psychology and neuroscience such as discussed in this paper. By organising three practical applications of the cloud of research-based theoretical knowledge discussed into a single matrix, we have created a framework to guide the next phase of the design-research, with the guiding question becoming: “How can emerging technologies be used to effectively and efficiently collect and organise data in ways that will inform the questions in the fold-back matrix in HEO quality processes?” We now pose the next design-research question with the belief that new technologies do in fact make the objective possible and will conclude this paper by illustrating potential ways forward through a discussion of our early work in repurposing two new, but well developed, applications. This discussion is based on the preliminary work of a further design jam (Strickdorn & Schneider, 2010) carried out by our design team in conjunction with colleagues from our university’s Teaching and Learning Centre who support quality enhancement across the university as part of a structural adjustment process. It is presented as a design hypothesis to indicate that the sorts of expanded and expansive quality processes we propose in this paper are now possible, and further design-research work into their implementation is now timely. The experience API: Learning everywhere, anytime Contemporary theories of learning show that learning is a complex and highly social activity. Learning occurs everywhere, across many different formal and informal settings and contexts. During this process the learner undertakes a variety of activities often directed to a specific goal or goals. They read or watch something, they practice a skill or technique, they discuss an idea with a colleague, they complete a quiz or written piece, they work in a team to complete a task or they create an artefact. As discussed above, however, current quality systems focus almost exclusively on the activity of the teaching staff (Pitman, 2014) and are not able to account for this broader student engagement (Coates, 2005). The xAPI suggests the potential to take an expanded account of student activity, and potentially to keep building an account beyond graduation. The primary function of the xAPI is to link activities to artefacts, and link them all to abstractions. The application can be run from a smart phone, tablet or computer and can create a learning record store for an individual, which can also be aggregated at a cohort, course or institutional level. It provides a more automated, timely and detailed ways of tracking, recording, managing and visualising learner activity. In such a learning records store, our hypothesis is that high quality learning will be indicated by evidence of a positive change in the learner’s skills, knowledge and dispositions. The xAPI is a platform independent and extensible methodology designed to track and record a user’s experiences with internet based content and associated learning activities. It is a successor to the Sharable Content Object Reference Model (SCORM), which is a set of standards and specifications for web-based learning or e-learning. SCORM has many limitations including being dependent on a web browser and a learning management system, being able to record only very limited types of activities, and being limited to tracking individual users. xAPI retains the capacity to chart experiences occurring inside traditional learning management systems (LMS), such as Moodle. It can also capture experiences on other internet platforms (e.g., social media), and with ongoing development, non-computer-based activities (e.g., undertaking a workshop). Australasian Journal of Educational Technology, 2016, 32(2). 25 ascilite Figure 2. Generating learning experiences using xAPI As summarised in Figure 2, the xAPI captures the learner’s profile and learning experiences and stores them in a database known as a Learning Records Store (LRS). The main role of the LRS is to authenticate and store the incoming statements that are used as evidence of learning, and then retrieve the data when requested by other systems. At its simplest level a statement looks like: , with , in . A collection of these statements would constitute a learning experience and evidence of learning. The xAPI is most often designed as a cloud-based datastore capable of collecting users’ learning experiences in both formal courses as well as participation in short courses, conferences, competency training and other activities. The datastore allows for the easy adding of data from a variety of sources such as LMS, individual learner input, mobile applications (e.g., conference apps) and other Internet connected devices (e.g., simulations). This input is summarised and provided as a dashboard. To test this capacity of the xAPI our design team has been collecting records of our own professional learning through our design-research work making use of the xAPI statement format, and digitally collecting samples of our work as evidence as diverse as photos of designs drawn on whiteboards, to notes form meetings with colleagues, to copies of literature reviews. Recognising that we are hardly a random sample, and that our design team is also a very small sample, what we have noted in our auto-ethnographic study is that unlike LMS systems organised around courses, the records of the LRS in the xAPI relate discreet learning activities to the learner, the result, and the context. As such, we find it responsive to contemporary theories of learning as it is able to take into account the role of the learner activity and context in learning, and does not embed an understanding that quality learning is driven only by the activity of the teacher. By comparing the records collected in this process to our own performance and development review documentation from previous years, we found that the technology led to us expanding what we were counting as professional learning. We were able to construct much richer accounts of the value of the activity we had engaged in, shifting from statements such as “completed course in X”, to statements such as “applied concept X working in context Y to achieve outcome Z”. While the accounts created did sound much like a good employment application, the notable aspect was that the technology supported collection of relevant data and evidence as it was created, and allowed it to be readily searched. In aggregated form, the data collected in this format has the potential to provide significant fold-back on how students are doing their learning, rather than simply feedback on how students feel about the activity of their teachers and institutions. It is a dataset suitable for interrogation via the framework set out in our fold-back matrix. Further, through being organised around a user profile, it also allows for a longer-term understanding of learner engagement to be developed, and so has the potential to provide meaningful data Australasian Journal of Educational Technology, 2016, 32(2). 26 ascilite across all of the cycles of value creation developed in the fold-back matrix. It may be used by graduates, for example, as an ongoing record of professional learning and simultaneously expand the understanding of how graduates are using the knowledge and skills developed in a course. Dedoose: Capturing and mixing multiple channels A major challenge to overcome in drawing together multiple channels and creating fold-back is the challenge of data inundation. Putting aside the challenges of methodology, the cost of working through large qualitative and mixed data sets is potentially prohibitive and a major reason why proxies such as student experience surveys are so common despite their inherent flaws. However, technologies that may reduce both the methodological and cost challenges are beginning to emerge. An example is Dedoose (2014), an application built to support research, but with potential to be re-purposed to support quality processes in HEOs. Dedoose is an application that supports qualitative and mixed-methods analysis of data in a variety of media including text, audio and video files. It is web-based and has been developed on a pay-as-you-use business model, eliminating high license costs for the casual user. Qualitative analysis is carried out through the coding of data excerpts. This requires a degree of research skill but the app greatly speeds the process compared to manual coding. Two significant features of Dedoose that allow for mixed- methods analysis within quality processes are the capacity to add multiple descriptors to each data source, and the capacity to give a numerical weighting to the qualitative coding. Recent research using Dedoose shows it can help reduce methodological and cost/time challenges (Kyte, Ives, Draper, Keeley, & Calvert, 2013; Young & Jaganath, 2013). The descriptor capacity of Dedoose, for example, may be used to add information such as age, gender, or previous study to a data source. This mixed-methods approach allows the rapid visualisation of qualitative data, along side quantitative data. Records generated and artefacts collected through LRS of the xAPI, for example, can be qualitatively interrogated for evidence relating to questions in the fold-back framework. Qualitative coding of artefacts with reference to the evaluative questions set out in the fold-back matrix can then be quickly organised using the descriptor data. This allows accounts of the learning experience of different types of students to be rapidly described and interpreted. This is important for how institutions enhance quality. For example, current quality data such as that from a graduate destination survey might reveal that the employment rate of graduates from a particular course is dropping at a significant rate. It may be possible to correlate this with, say, a decline in entry standards to the course. This is important information as it suggests a decline in the quality of the course due to a decline in the quality of students, but it is not information that supports the educational decisions that need to be made in response. It is not information that supports quality enhancement. Interrogating LRS data with the Dedoose application, however, can allow an account to be developed of how students from different prior learning backgrounds actually interact with the course. It can allow a picture to be built of how those students go about their learning, how they interact with the intellectual and social context, and how they apply new concepts, skills and dispositions. We have demonstrated the value of such phenomenographic analysis of course experiences, and the capacities of emerging technologies to support it, elsewhere (Leonard & Roberts, 2014, 2015) and contend that, if well designed, approaches such as those developed in that research can be scaled to institutional quality processes. A feature of Dedoose that expands the phenomenographic potential is its capacity to assign a weighting when coding data excerpts. This has many potential uses. In test coding of student-produced web pages, for example, we used the rating to indicate which cycle of value the student was seeing for a concept: immediate; potential; applied; realised; or reframing. Dedoose offered a range of options for visualising the variations we were finding, allowing us to readily see the ways in which different concepts were being understood, and to visually cross match the variation back to basic group demographics within the course. We will provide a fuller account of the possibilities of this type of visualisation within evaluation in future publications, but for the purposes of the present illustrative sketch, we have provided an example in Figure 3. Figure 3 shows the average code weighting applied in examining a collection of student work using Dedoose. Three codes are presented simultaneously and related to students’ highest previous degree. Within the very small sample used to produce this plot (the work being a design sketch rather than research) it was apparent that students with higher degree research backgrounds saw a higher-level value in research informing their professional practice, but they saw comparatively low-level value in collaboration. Students with bachelors degrees, in contrast, saw a higher level of value in the use of media - although on the surface, it seems likely that this may be more closely associated with age than prior study. Our purpose here though Australasian Journal of Educational Technology, 2016, 32(2). 27 ascilite is not to demonstrate a specific relationship, but to point to the sorts of examination a tool such as Dedoose makes possible. Figure 3. Average code-weighting plot from Dedoose Ongoing design-based research New technology provides a significant opportunity to enhance quality and improve the value of learning within higher education, but this use of technology has been relatively unexplored in the broader scholarship educational technology. This needs redressing as an increasingly data rich environment demands a clear understanding of which data is most useful, or is useful at all, and in what ways. It is necessary, for example, to avoid focussing on data that is easy to collect if that data has little meaning. It is similarly important to differentiate between data that might assist with an educational institution’s marketing and data that might inform decisions about students’ learning, which is not necessarily the same thing. In the form of a matrix of evaluative questions drawing together practical applications of contemporary understandings of human learning from psychology and neuroscience, this paper has proposed a framework for an expanded and expansive understanding of quality in higher education. To illustrate how new technologies might support such an expansive approach, the paper has then provided initial design sketches on how two emergent applications could be used to implement the framework. The change and potential we suggest is similar to that which occurred in the 1970s when off-the-shelf statistical software allowed the expansion of correlative statistical analysis by bringing within the reach of most institutions the use of computers to process large data sets. Technologies such as xAPI and Dedoose offer a similar step-change for qualitative and mixed data sets and, therefore, allow us to begin designing quality processes that make use of much larger data sets of this type. In turn, the expanded data opportunities allow quality processes to develop accounts of student activity, of their interaction with networks, communities, and contexts, and to understand their application of key skills and concepts. Taking advantage of this emerging capacity has potential to significantly improve the understanding of the learning activities that are occurring in higher education, and hence to improve the quality of those activities. How we best make use of this emerging capacity is worthy of ongoing design-research. We conclude, however, by noting that the technologies provide only a capacity. There remains a need for skilled educators to identify important data and interpret the results, to mix the fold-back, and the role of the educator must remain a key focus in further research. Australasian Journal of Educational Technology, 2016, 32(2). 28 ascilite Once the most useful data is identified it must also be recognised that better information is only valuable when used as part of an effective formative evaluation process, and that poorly designed evaluation systems may even have the adverse and unintended effect of impeding learning. The potential value of more effective evaluation approaches that support rich and complex understandings of learning environments is therefore, substantial and as such warrants further design-research work. Acknowledgements This paper was developed through work within the University of Canberra’s SAFFIRE project funded by the Australian Government. We thank our colleagues at the University of Canberra who participated in our design jams. References Ablin, J. L. (2008). Learning as problem design versus problem solving: Making the connection between cognitive neuroscience research and educational practice. Mind, Brain & Education, 2(2), 52-54. doi:10.1111/j.1751-228X.2008.00030.x Advanced Distributed Learning. (2014). Training & learning architecture (TLA): Experience API (xAPI). Retrieved from http://www.adlnet.gov/tla/experience-api/ Akerlind, G. S. (2005). Academic growth and development: How do university academics experience it? Higher Education, 50(1), 1-32. doi:10.1007/s10734-004-6345-1 Allais, S. (2011). ‘Economics imperialism’, education policy and educational theory. Journal of Education Policy, 27(2), 253-274. doi:10.1080/02680939.2011.602428 Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in education research? Educational Researcher, 41(1), 16-25. doi:10.3102/0013189X11428813 Ballantyne, R., Borthwick, J., & Packer, J. (2000). Beyond student evaluation of teaching: Identifying and addressing academic staff development needs. Assessment & Evaluation in Higher Education, 25(3), 221-236. doi:10.1080/713611430 Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of Learning Sciences, 13(1), 1-14. doi:10.1207/ s15327809jls1301_1 Bennett, L., & Nair, C. S. (2010). A recipe for effective participation rates for web-based surveys. Assessment & Evaluation in Higher Education, 35(4), 357-365. doi:10.1080/02602930802687752 Berliner, D. C. (2009). MCLB (much curriculum left behind): A US calamity in the making. Educational Forum, 73(4), 284-296. doi:10.1080/00131720903166788 Boysen, G. A., Kelly, T. J., Raesly, H. N., & Casner, R. W. (2014). The (mis)interpretation of teaching evaluations by college faculty and administrators. Assessment & Evaluation in Higher Education, 39(6), 641-656. doi:10.1080/02602938.2013.860950 Burrows, S., & Shortis, M. (2011). An evaluation of semi-automated, collaborative marking and feedback systems: Academic staff perspectives. Australasian Journal of Educational Technology, 27(7), 1135- 1154. Retrieved from http://ajet.org.au/index.php/AJET/article/view/909 Campbell, D. T. (1976). Assessing the impact of planned social change, Paper #8. Paper presented at the Occasional Paper Series, Darthmouth College, Hanover, NH. Carrell, S. E., & West, J. E. (2010). Does professor quality matter? Evidence from random assignment of students to professors. Journal of Political Economy, 118(3), 409-432. doi:10.3386/w14081 Cheung, W. S., & Hew, K. F. (2011). Design and evaluation of two blended learning approaches: Lessons learned. Australasian Journal of Educational Technology, 27(8), 1319-1337. Retrieved from http://ajet.org.au/index.php/AJET/issue/view/12 Coates, H. (2005). The value of student engagement for higher education quality assurance. Quality in Higher Education, 11(1), 25-36. doi:10.1080/13538320500074915 Crockett, L., Jukes, I., & Churches, A. (2011). Literacy in not enough: 21st-century fluencies for the digital age. Kelowna, BC & Thousand Oaks, CA: 21st Century Fluency Project & Corwin. Davies, P. (2006). How can we recognise them? In J. Meyer, & R. Land (Eds.), Overcoming barriers to student understanding : Threshold concepts and troublesome knowledge. Abingdon, Oxon: Routledge. Dedoose. (2014). Dedoose. Retrieved from http://www.dedoose.com Australasian Journal of Educational Technology, 2016, 32(2). 29 ascilite Dommeyer, C. J., Baum, P., Hanna, R. W., & Chapman, K. S. (2004). Gathering faculty teaching evaluations by in-class and online surveys: Their effects on response rates and evaluations. Assessment & Evaluation in Higher Education, 29(5), 611-623. doi:10.1080/02602930410001689171 Elassy, N. (2015). The concepts of quality, quality assurance and quality enhancement. Quality Assurance in Education, 23(3), 250-261. doi:10.1108/QAE-11-2012-0046 Engestrom, Y. (2001). Expansive learning at work: Toward an activity theoretical reconceptualisation. Journal of Education and Work, 14(1), 133-155. doi:10.1080/13639080020028747 Engeström, Y. (2006). Activity theory and expansive design. In S. Bagnara, & G. C. Smith (Eds.), Theories and practice in interaction design. (pp. 3-23). Mahwah, NJ: Lawrence Erlbaum Associates Publishers. Filippakou, O., & Tapper, T. (2008). Quality assurance and quality enhancement in higher education: Contested territories? Higher Education Quarterly, 62(1‐2), 84-100. doi:10.1111/j.1468- 2273.2008.00379.x Hargreaves, A., & Fullan, M. (2012). Professional capital: Transforming teaching in every school. New York, NY: Teachers College Press. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81- 112. doi:10.3102/003465430298487 Hattie, J., & Yates, G. (2014). Visible learning and the science of how we learn. New York, NY: Routledge. Hayes, D., Lingard, B., Mills, M., & Christie, P. (2006). Teachers & schooling making a difference: Productive pedagogies, assessment and performance. Crows Nest, NSW: Allen & Unwin. Hutchins, E. (2005). Material anchors for conceptual blends. Journal of Pragmatics, 37(10), 1555-1577. doi:10.1016/j.pragma.2004.06.008 Hutchins, E. (2010). Cognitive ecology. Topics in Cognitive Science, 2(4), 705-715. doi:10.1111/j.1756- 8765.2010.01089.x Kane, D., Williams, J., & Cappuccini-Ansfield, G. (2008). Student satisfaction surveys: The value in taking an historical perspective. Quality in Higher Education, 14(2), 135-155. doi:10.1080/13538320802278347 Kay, R. H., & Knaack, L. (2008). A multi-component model for assessing learning objects: The learning object evaluation metric (LOEM). Australasian Journal of Educational Technology, 24(5), 574-591. Retrieved from http://ajet.org.au/index.php/AJET/issue/view/33 Kuh, G. D. (2003). What we're learning about student engagement from NSSE. Philadelphia, PA: Taylor & Francis Inc. Kyte, D., Ives, J., Draper, H., Keeley, T., & Calvert, M. (2013). Inconsistencies in quality of life data collection in clinical trials: A potential source of bias? Interviews with research nurses and trialists. PLoS ONE, 8(10), e76625. doi:10.1371/journal.pone.0076625 Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. New York, NY: Cambridge University Press. Leckey, J., & Neill, N. (2001). Quantifying quality: The importance of student feedback. Quality in Higher Education, 7(1), 19-32. doi:10.1080/13538320120045058 Leonard, S. N., & Roberts, P. (2014). Performers and postulates: the role of evolving socio-historical contexts in shaping new teacher professional identities. Critical Studies in Education, 55(3), 303-318. doi:10.1080/17508487.2014.904808 Leonard, S. N., & Roberts, P. (2015). No time to think: The impact of the logics of new public management on teacher pre-professional learning. Journal of Education Policy, 30(6), 1-19. doi:10.1080/02680939.2015.1047801 Light, A., & Akama, Y. (2012). The human touch: Participatory practice and the role of facilitation in designing with communities. Paper presented at the 12th Participatory Design Conference, New York, NY. Lingard, B. (2011). Policy as numbers: Ac/counting for educational research. Australian Educational Researcher, 38(4), 355-382. doi:10.1007/s13384-011-0041-9 Lingard, B., & Sellar, S. (2013). 'Catalyst data': Perverse systemic effects of audit and accountability in Australian schooling. Journal of Education Policy, 28(5), 634-656. doi:10.1080/02680939.2012.758815 Marshall, J. D. (1999). Performativity: Lyotard and Foucault through Searle and Austin. Studies in Philosophy and Education, 18(5), 309-317. doi:10.1023/a:1005272607727 Australasian Journal of Educational Technology, 2016, 32(2). 30 ascilite Marton, F. (1986). Phenomenography: A research approach to investigating different understandings of reality. Journal of Thought, 21, 289-303. Marton, F., & Tsui, A. B. M. (2004). Classroom discourse and the space of learning. Mahwah, NJ: Lawrence Erlbaum Associates. Miller, J. E., & Seldin, P. (2014). Changing practices in faculty evaluation. Washington, DC: American Association of University Professors. Murphy, E., & Rodriguez-Manzanares, M. A. (2008). Using activity theory and its principle of contradictions to guide research in educational technology. Australasian Journal of Educational Technology, 24(4), 442-257. Retrieved from http://ajet.org.au/index.php/AJET/issue/view/34 Nair, C. S., Mertova, P., & Pawley, D. (2010). Quality in action: Closing the loop. Quality Assurance in Education, 18(2), 144-155. doi:10.1108/09684881011035358 Nelson, K. J., Smith, J. E., & Clarke, J. A. (2012). Enhancing the transition of commencing students into university: An institution-wide approach. Higher Education Research & Development, 31(2), 185- 199. doi:10.1080/07294360.2011.556108 Newton, G., & Martin, E. (2013). Blooming, SOLO taxonomy, and phenomenography as assessment strategies in undergraduate science education. Journal of College Science Teaching, 43(2), 78. doi:10.2505/4/jcst13_043_02_78 Nowell, C., Gale, L. R., & Handley, B. (2010). Assessing faculty performance using student evaluations of teaching in an uncontrolled setting. Assessment & Evaluation in Higher Education, 35(4), 463-475. doi:10.1080/02602930902862875 Phillips, R., Kennedy, G., & McNaught, C. (2012). The role of theory in learning technology evaluation research. Australasian Journal of Educational Technology, 28(7), 1103-1118. Retrieved from http://ajet.org.au/index.php/AJET/issue/view/7 Pitman, T. (2014). Reinterpreting higher education quality in response to policies of mass education: The Australian experience. Quality in Higher Education, 20(3), 348-363. doi:10.1080/13538322.2014.957944 Power, M. (2009). The risk management of nothing. Accounting, Organizations & Society, 34(6/7), 849- 855. doi:10.1016/j.aos.2009.06.001 Ramsden, P. (1991). A performance indicator of teaching quality in higher education: The course experience questionnaire. Studies in Higher Education, 16(2), 129-150. doi:10.1080/03075079112331382944 Saarinen, T. (2010). What I talk about when I talk about quality. Quality in Higher Education, 16(1), 55- 57. doi:10.1080/13538321003679507 Sandoval, W. A. (2004). Developing learning theory by refining conjectures embodied in educational designs. Educational Psychologist, 39(4), 213-223. doi:10.1207/s15326985ep3904_3 Sandoval, W. A., & Bell, P. (2004). Design-based research methods for studying learning in context: Introduction. Educational Psychologist, 39(4), 199-201. doi:10.1207/s15326985ep3904_1 Scott, G., Shah, M., Grebennikov, L., & Singh, H. (2008). Improving student retention: A University of Western Sydney case study. Journal of Institutional Research, 14(1), 1-23. Retrieved from http://www.aair.org.au/articles/volume-14-no-1/14-1-improving-student-retention-a-university-of- western-sydney-case-study Shah, M. (2012). Ten years of external quality audit in Australia: Evaluating its effectiveness and success. Assessment and Evaluation in Higher Education, 37(6), 761-772. doi:10.1080/02602938.2011.572154 Shum, S. B., & Ferguson, R. (2011). Social learning analytics. [Available as: Technical Report KMI-11- 01] (Retrieved from http://kmi.open.ac.uk/publications/pdf/kmi-11-01.pdf). London: Knowledge Media Institute, The Open University. Siemens, G. (2014). The Journal of Learning Analytics: Supporting and promoting learning analytics research. Journal of Learning Analytics, 1(1), 3-5. Retrieved from http://epress.lib.uts.edu.au/journals/index.php/JLA/article/view/3908/4010 Strickdorn, M., & Schneider, J. (2010). This is service design thinking. Amsterdam: BIS Publishers. Symons, R. (2006). Listening to the student voice at the University of Sydney: Closing the loop in the quality enhancement and improvement cycle. Paper presented at the Australian Association for Institutional Research Forum, Coffs Harbour, NSW. Retrieved from http://www.aair.org.au/app/webroot/media/pdf/AAIR Fora/Forum2006/Symons.pdf Usher, R. (2006). Lyotard’s performance. Studies in Philosophy & Education, 25(4), 279-288. doi:10.1007/s11217-006-9009-z Australasian Journal of Educational Technology, 2016, 32(2). 31 ascilite Verenikina, I. (2010). Vygotsky in twenty-first-century research. In J. Herrington, & C. Montgomerie (Eds.), Proceedings of EdMedia: World Conference on Educational Media and Technology 2010, Totonto, Canada, 16-25. Retrieved from http://www.editlib.org/p/34614/ Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes (M. Cole, Trans.). Cambridge, MA: Harvard University Press. Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge, NY: Cambridge University Press. Wenger, E., Trayner, B., & Laat, M. D. (2011). Promoting and assessing value creation in communities and networks: A conceptual framework. Heerlen: Ruud de Moor Centrum. Retrieved from http://wenger-trayner.com/documents/Wenger_Trayner_DeLaat_Value_creation.pdf Wiggins, G. P., & McTighe, J. (2011). The understanding by design guide to creating high-quality units. Alexandria, VA: ASCD. Wong, V. Y.-Y. (2012). An alternative view of quality assurance and enhancement. Management in Education, 26(1), 38-42. doi:10.1177/0892020611424608 Young, S. D., & Jaganath, D. (2013). Online social networking for HIV education and prevention: A mixed methods analysis. Sexually Transmitted Diseases, 40(2). doi:10.1097/OLQ.0b013e318278bd12 Corresponding author: Simon Leonard, simon.leonard@canberra.edu.au Australasian Journal of Educational Technology © 2016. Please cite as: Leonard, S. N., Fitzgerald, R. N., & Bacon, M. (2016). Fold-back: Using emerging technologies to move from quality assurance to quality enhancement. Australasian Journal of Educational Technology, 32(2), 15-31.