phillips.pdf Australasian Journal of Educational Technology 2012, 28(7), 1103-1118 The role of theory in learning technology evaluation research Rob Phillips Murdoch University Gregor Kennedy The University of Melbourne Carmel McNaught The Chinese University of Hong Kong This paper attempts a fundamental analysis of the nature of research into e-learning and the role that theory plays in this. We examine ‘research’ in broad terms, and the nature of phenomena in general. We identify that e-learning is an artificial phenomenon, and that research approaches need to be cognisant of the design elements in e-learning, and the cyclical nature of e-learning development. We identify various desired research outcomes which are appropriate at each stage of the e- learning lifecycle, and argue that studies of e-learning involve a mixture of evaluation and research. We discuss e-learning evaluation research in the context of different disciplinary and interdisciplinary research approaches, recognising that there is no one ‘right’ way to do e-learning evaluation research. However, we recognise that there is a varying mixture of a ‘search for fundamental understanding’ and ‘consideration of use’ in e- learning evaluation research. We use these considerations to discuss the role of theory in educational research, and, in particular, in e-learning evaluation research, before applying the preceding arguments to the e-learning lifecycle, identifying five different forms of evaluation research. Introduction The application of theory in learning technology research has had a troubled history. We contend that strong, theory-based research in learning technology has been difficult for three reasons. First, as an emerging field, e-learning or learning technology has attracted a diverse set of individual scholars who seek to apply research approaches from their own ‘discipline-of-origin’ to the field of learning technology. Second, learning technology is a multifaceted phenomenon involving design, development, practice and research; and academics and scholars have often focused on developing and implementing technologies for their teaching practice rather than conducting theory-based research. Third, scholarly inquiry in learning technology has often concerned itself with atheoretical evaluations in local learning and teaching contexts – did it work? – at the expense of theory-based research. In this paper we contend that these form barriers to robust, theory-based learning technology research We address these issues in subsequent sections, before presenting an approach to research in learning technology that we feel is both scholarly and pragmatic. 1104 Australasian Journal of Educational Technology, 2012, 28(7) However, first we need to clarify our terminology. Over the last decades, terms to describe the use of computers to assist people to learn have ebbed and flowed in terms of popularity. Terms such as educational technology, learning technology, technology- enhanced learning, computer-facilitated learning, and e-learning have been in use in the last decade, without clear definitions of their scope and meaning. All of these terms are problematic, because they valorise technology over learning and infer that learning can be delivered through technology, discounting the personal, cognitive aspects of learning. Given this, we agree with Friesen that e-learning has “come to represent a useful shorthand for a range of different orientations to … the use of technologies in education and learning.” (2009, p.4). We choose to use, as one among many, the definition of Littlejohn and Pegler: “the process of learning and teaching with computers and other associated technologies, particularly through use of the Internet” (2007, p.15). In this definition, the focus, as it should be, is on learning – learning facilitated through technological tools, and using technology to address an educational need. In this paper, we will mix the use of the terms learning technology and e- learning, using e-learning typically as an adjective to describe aspects of the learning environment, including its research. The ‘thesis’ of this article is derived from our recent book, Evaluating e-learning: Guiding research and practice (Phillips, McNaught & Kennedy, 2011). This article seeks to complement the book’s orientation towards new practitioners and more practical matters, by advancing a strong emphasis on theoretical issues to influence the perspectives of experienced researchers. Diverse approaches to learning technology research Learning technology is primarily a branch of the field of education, but it also brings in influences from other fields, including computer engineering, information technology, design and media studies. It is interdisciplinary in that it seeks to combine and explore the interconnections between new and different approaches from different fields and specialisations; it is multidisciplinary in that it simultaneously tries to respect the multiplicity of differences that can separate one research approach from another. (Friesen, 2009, p. 12). While learning technology is developing as a field, often individual academics have come to e-learning from established disciplines, and have tended to bring with them their particular, discipline-specific research traditions. Shulman (1988) contended that What distinguishes disciplines from one another is the manner in which they formulate their questions, how they define the content of their domain and organize that content conceptually, and the principles of discovery and verification that constitute the ground rules for creating and testing knowledge in their fields. (Shulman, 1988, p. 5). Given that the emerging field of learning technology draws on numbers of disciplines and attracts multidisciplinary researchers, it is worth considering the characteristics of various disciplinary research approaches, to see which can be appropriately applied to learning technology research. Phillips, Kennedy and McNaught 1105 Disciplinary differences between types of academic work can be distinguished across two dimensions: pure versus applied, and hard versus soft (Becher, 1989; Biglan, 1973; Jones, Zenios & Griffiths, 2004). The distinction between pure (or basic) research and applied research became the accepted way of classifying research in the second half of the 20th century. The primary aim of pure research was conceived as seeking fundamental understandings of a natural phenomenon, while applied research was more focused on solving practical problems. The inter- and multidisciplinary nature of e-learning research indicates that many research approaches may be appropriate, but that some elements may be inappropriate. Stokes (1997) has argued that the pure–applied distinction is too narrow and does not consider how different types of research might be used. He proposed a two- dimensional model for classifying research that he called Pasteur’s quadrant (Table 1). One dimension of Pasteur’s quadrant classified research in terms of the degree to which it reflected a quest for fundamental understanding, while the second dimension to the research classification classified research by its ‘consideration of use’. Stokes argued that Neils Bohr’s work on atomic structure had no consideration of use, but sought fundamental understanding (Quadrant 1 in Table 1). However, he argued that microbiologist Louis Pasteur’s work was inspired by both consideration of use and a quest for fundamental understanding (Quadrant 2 in Table 1). On the other hand, Thomas Edison’s inventions applied existing understanding to develop new tools and techniques to solve real-world problems, without any intention of extending that understanding (Quadrant 4). One might question whether research in the third quadrant, which has no consideration for use and no quest for fundamental understanding, has any value, or, indeed, is academic research at all. Stokes (1997) characterised this quadrant as “research that systematically explores particular phenomena without having in view either general explanatory objectives or any applied use to which the results will be put” (italics in original; p. 74). The dedicated activities of amateur astronomers and birdwatchers fit into this category, and this type of ‘research’ may ultimately prove to be of considerable value to other researchers, in other quadrants, at a later stage. Table 1: Pasteur’s quadrant Consideration of use 1. Pure basic research (Bohr) 2. Use-inspired basic research (Pasteur)Quest forfundamental understanding 3.Amateur ‘researchers’ (e.g. birdwatchers) 4. Pure applied research (Edison) Stokes’ (1997) work challenged two assumptions of the traditional pure–applied dichotomy: (i) that pure research sought fundamental understanding while applied research did not; and (ii) that basic research always preceded applied research. Another lens through which to look at academic research is Boyer’s (1990) four different types of scholarship. This analysis is also based on a critique of the post-war emphasis on pure research, which Boyer (1990) called the scholarship of discovery, that is, research that pursues new knowledge and fundamental understanding. In addition to the scholarship of discovery, Boyer (1990) proposed three other types of scholarship: 1106 Australasian Journal of Educational Technology, 2012, 28(7) the scholarship of integration, the scholarship of application and the scholarship of teaching. The scholarship of integration involves connecting knowledge and discovery into larger patterns and contexts. This includes interdisciplinary and multidisciplinary research, at the “boundaries where fields converge” (p. 19). This paper represents an example of the scholarship of integration. The scholarship of application involves engagement in problems that affect individuals, institutions, and society, and asks questions such as: “How can knowledge be responsibly applied to consequential problems? How can it be relevant to society?”. The scholarship of teaching is perhaps the hardest to conceptualise. It is not scholarship about teaching, but the scholarship o f teaching. Hutchings and Shulman (1999) distinguished between good teaching and the scholarship of teaching in that the latter gathers evidence; is informed by current ideas about the field, and teaching in that field; and invites “peer collaboration and review” (p. 13). Further, the scholarship of teaching is public, “open to critique and evaluation and in a form that others can build on” (p. 13); and it involves inquiry into “issues of student learning” (p. 13). Boyer’s four scholarships have some overlaps with Stokes’ schema. The scholarship of discovery and the scholarship of application are directly analogous to Stokes’ ‘fundamental understanding’ and ‘consideration of use’ components, respectively, which are conceived of as separate dimensions by Stokes (1997). The scholarship of integration cuts across Stokes’ boundaries, and the scholarship of teaching builds on the other scholarships. As Hutchings and Shulman (1999) explained, scholarship of teaching “is a special case of the scholarship of application and engagement, and frequently entails the discovery of new findings and principles.” (p. 15). Using these various lenses with which to view research, we can conclude that learning technology research can be inspired both by use (in Stokes’ terms) and by quest for fundamental understanding, as well as by Boyer’s four scholarships. Learning technology investigations often study the activities of learners in a specific learning environment and are aimed at better understanding how technology can be applied and used. Learning technology investigations can also seek to further our understanding of how students learn with technology. Learning technology as a multifaceted phenomenon When thinking about scientific or systematic research Simon (1969) distinguished between the natural sciences, which are concerned with discovering how natural phenomena work – in fields such as physics, biology, and anthropology – and artificial sciences. Artificial sciences seek to design artefacts, understand and reflect on them, and ultimately improve their design and use, for example in the fields of engineering and architecture, designing aeroplanes, bridges or buildings. However, designed artefacts do not have to be physical ‘things’ – they may also be less tangible, such as a computer program or an e-learning environment. Events, such as interactions between people, are also artificial phenomena. In an educational context, a school or university class is an event phenomenon: a coming- together of teacher, learners and various resources in a particular setting, usually following some design activity. Each class is potentially different, and any understanding derived from a given class may be different from an understanding derived from a different class. This highlights an important distinction. With natural Phillips, Kennedy and McNaught 1107 phenomena, there are many instances of the same phenomenon, and this can be generalised through observation and measurement. With event phenomena, the combination of time, place and actors implies that there are single instances of multiple phenomena, making generalisation difficult. Finally, research into designed phenomena has an extra element not present when researching natural phenomena. Research in the artificial sciences needs to consider the way in which a ‘manufactured’ artefact functions, and whether it functions as designed. With natural phenomena, researchers have to take them as they are; but with designed phenomena, there is potential to improve the phenomenon through its design. Thus, research into designed phenomena is not only concerned with the behaviour of that phenomenon, but also with the design and functionality of the artefact which represents the phenomenon. In Stokes’ terms, ‘consideration of use’ is of comparable importance to ‘quest for understanding’. An implication of this discussion is that we need to think carefully about the phenomenon of e-learning before we start to study that phenomenon. As an artificial phenomenon, learning technology results from a design activity, where the outcome of the design activity is an e-learning artefact. We take a broad view of the interpretation of artefact to mean both tools developed using information and communication technologies (ICTs) and learning tasks designed through these tools. Once an e- learning artefact has been developed, it needs to be embedded into a designed learning environment (an event phenomenon) which specifies the interactions between learners, teachers and resources to meet a defined educational need. A consequence of this description is to consider the creation of e-learning artefacts and e-learning environments as proceeding according to a lifecycle, arising from a complex, multidisciplinary process, with multiple design, develop, implement and evaluate cycles (Duncan, 1996; England & Finney, 1999; Howell, 1992; Thornton & Phillips, 1997). One characterisation of the process is shown in Table 2, which draws from the work of Richey, Klein and Nelson (2004), and Reeves and Hedberg (2003). Table 2 defines seven stages in an idealised e-learning lifecycle, together with the typical activities and outcomes sought at each stage. The process starts with an analysis of the problem, identifying needs and defining the requirements of the solution. After analysis of the problem, the first cycle starts with the design of the e-learning artefact and associated documentation. That design should be evaluated to see if it is fit for purpose and how it could be improved. The second cycle begins with a refinement of the design and then the development of the e-learning artefact to a stage where it can be trialled. This initial trial may lead to a revision of the problem analysis – the proposed solution might not work well. Alternatively, if the design was well-grounded, it may lead to a process of designing an e-learning environment by embedding the e-learning artefact into a context defined by the designed learning outcomes and designed learning tasks. This e-learning environment is then developed as a pilot (cycle 3) and formatively evaluated. A subsequent cycle of revision and formative evaluation will lead into a full trial, with learners using the e-learning environment on a live system. Successful completion of cycle 4 indicates that the e-learning environment is ready to be used in a standard teaching situation. Refinement and revision is expected to be minimal at this stage. Cycles 5 and 6 correspond to the live use of a mature e-learning environment. 1108 Australasian Journal of Educational Technology, 2012, 28(7) While there is still a formative, continual-improvement component, the focus turns to the effectiveness of the e-learning environment, investigating learning processes and outcomes. Table 2: Components of an idealised e-learning lifecycle Cycle Lifecycle stage Development activity Outcome sought 0 Analysis of problem Analyse the learning problem Problem definition 1 Design e-learning artefact Design the e-learning artefact and document the design Design validated 2 Prototype e-learning artefact Refine the design, develop the e- learning artefact and conduct an initial trial Functioning artefact Areas for improvement identified 3 Design e-learning environment and conduct pilot study Design and develop an e-learning environment which embeds the e- learning artefact and pilots it E-learning environment functions as designed 4 Refine e-learning environment and conduct full trial Revise the e-learning environment and conduct a full trial with learners Areas for improvement identified 5 Evaluation research on mature system Revise the e-learning environment, deploy it to learners and start to understand how it works (learning processes) Initial understanding of how learners interact with the learning environment 6 Evaluation research on mature system Revise the e-learning environment, deploy it to learners and refine understanding about how it works (learning outcomes) Refined understanding of how learners interact with the learning environment It should be clear from this characterisation that the phenomenon of e-learning continually evolves and different outcomes are appropriate throughout this evolution. Any research activity associated with e-learning needs to be cognisant of, and appropriate to, the state of the phenomenon in the e-learning lifecycle. Evaluation and research in learning technology The e-learning lifecycle, as we have portrayed it, has a strong evaluative aspect but it also has a focus on research, particularly as a learning environment matures. This section explores the distinction between these activities, evaluation and research, in the context of a learning technology inquiry. The term ‘research’ is used in a variety of ways in academic discourse but, despite this, its meaning is rarely questioned and often it is regarded as universally or even implicitly understood. In most universities an institutional bureaucracy supports research, and academic staff are rewarded for their ‘strength’ in research. However, there are many contexts in which research takes place and many different ways in which research is conducted. Unless this diversity of understanding is recognised, it is difficult to have a meaningful dialogue about it. This is particularly true in an emerging field such as e-learning. Our view of ‘research’ resonates with what Shulman (1988) termed disciplined inquiry. This can be distinguished “from other sources of opinion or belief [and] is conducted and reported in such a way that the argument can be painstakingly examined” (Cronbach & Suppes, 1969, p. 15). Shulman (1988) viewed academic research as that Phillips, Kennedy and McNaught 1109 which is disciplined, systematic, explicit and ethical, whose “data, arguments and reasoning [are] capable of withstanding careful scrutiny by another member of the scientific community” (1988, p. 5). Research is primarily involved with increasing our understanding of a phenomenon. This can be distinguished from evaluation, which has an explicit focus on gathering information to help make judgments about the value and worth of an object in order to inform decision-making. This is a somewhat vexed issue as some authors have argued that providing an explanation of outcomes is not within the remit of evaluation (e.g. Glass & Worthen, 1971). However, others, such as Oliver, Harvey, Conole and Jones (2007), have argued that evaluation and research studies can use similar methods to arrive at similar outcomes, but they can be distinguished by the role of theory in interpreting results, and in the way those outcomes are used. We argue that investigations in learning technology can have, and should have, both an evaluative and a research focus. Some inquiries in e-learning can be seen as research – gathering information to inform our understanding of how people learn using an e-learning artefact or environment – while others are more related to evaluation – gathering information to help make judgments about the value and worth of an e-learning artefact or environment. A study of the effectiveness of an e-learning environment may quite easily shed light on how learners engaged with the designed learning processes to achieve their results, or why some learners achieved at different levels, or how some learners used the learning environment to achieve a deeper understanding. While any of these findings could be seen as the outcomes of an evaluation study, they could equally be seen as legitimate outcomes of an educational research investigation. We use the term ‘evaluation research’ to simply capture the idea that investigations of e-learning will often involve a mix of evaluation and research activities that can be applied throughout the e-learning lifecycle. When it comes to learning technology investigations, there is typically an ebb and flow between making judgments about the e-learning environment and developing a greater understanding of learning in that environment. The role of theory in educational research Unfortunately, one of the consistent criticisms of educational research generally, and e- learning in particular, has been that it is often conducted without a strong theoretical basis (Reeves, 1993). What then is the role of theory in our characterisation of learning technology evaluation research? The characteristics of a theory are that it is derived from empirical evidence or from other theories; that it can provide a generalised explanation of a phenomenon to the accuracy of the evidence, sometimes based on a model, framework or analogy, and it can predict the behaviour of another instance of the phenomenon. Further, “it is necessary to have a clear view of the reliable range of each aspect of the theory” (Burkhardt, 2006, p. 124), and the assumptions which underpin it. “A strong theory provides an explanation of what is behind an array of observations” (Burkhardt, 2006, p. 130). In the pure sciences, theories are usually strong and generalisable, like the theory of gravity proposed by Newton. In other disciplines, theory is less strong. We alluded earlier to a focus in the Western world since the 1940s on basic versus applied research. Much thinking about research has been about research and 1110 Australasian Journal of Educational Technology, 2012, 28(7) development, with the implication that basic research is more valuable than, and is a precursor to, the less valuable applied research which results in the development of technology. The research then development approach is appropriate in disciplines where there is a strong theory which predicts behaviour. This notion that research precedes development was challenged by Stokes (1997), who argued that useful research is often preceded by practical development, especially in emerging fields (cf. the earlier discussion about Newton). For example, Burkhardt claims that “in medicine, theory is moderately weak” (2006, p. 131). The development of new drugs, for example, often starts with exploratory studies of natural substances. It is “only after exploratory clinical research … followed by dose-response and early safety trials” (Zaritsky, Kelly, Flowers, Rogers & O'Neill, 2003, p. 32) is successful that theory-based “large-scale comparative trials in medicine (often promoted as the ‘gold standard’ for research) occur” (Zaritsky, Kelly, Flowers, Rogers & O'Neill, 2003, p. 32). Burkhardt (2006) goes on to claim that “education is a long way behind medicine, let alone engineering, in the range and reliability of its theories” (p. 131). This is because educational environments are complex systems (Salomon, 1991), consisting of an array of interdependent individual and contextual variables, where one would typically observe ‘differences in patterns’, rather than ‘patterns of differences’, which are the goal of controlled experimental research (Rowe, 1996). Constructing broad, generalisable and predictive theories of these systems is difficult. However, while there may be fewer predictive and generalisable educational theories compared to some areas of the natural sciences (cf. Burkhardt, 2006; Flyvbjerg, 2004), this does not reduce at all the importance of theory in educational research. Because of this, in education theories need to be developed before they can be tested, and, “design plays a critical role in the development of theories” (Edelson, 2002, p. 106). However, this does not reduce the importance of attempting to develop a theoretical description of the phenomenon being studied, which can predict what will happen in a given context. In education, theory tries to come up with (testable) explanations of some aspect of learning. In addition to established theories of education and learning, there is a proliferation of models and conceptual frameworks in educational research. These are looser than theory, and are generally not encumbered by the need to offer predictability. Their central purpose is usually to expose, describe, categorise and make order of some phenomenon. Conceptual frameworks will often, by their very construction, highlight the important elements to take into account when considering some phenomenon. One such example is Herrington, Reeves and Oliver’s (2010) authentic learning framework. It provides order and highlights what is important if you want to adopt authentic approaches to teaching and learning. Similarly, the LEPO (Learning Environment, Processes, Outcomes) framework (Phillips, McNaught & Kennedy, 2011) is a conceptual framework which we use later in this paper in unpacking the nature of e-learning evaluation research. This conceptual framework sees learners and teachers interacting through learning tasks which are enacted in a learning environment: students attain learning outcomes by going through learning processes embedded in a learning environment. The LEPO framework provides a useful way of viewing the complexity of learning, but it has no predictive capabilities, nor is it backed up by specific evidence, other than a critical synthesis of the literature. Phillips, Kennedy and McNaught 1111 The conclusion that we draw from this discussion is that discovery research, such as that used when investigating natural phenomena, is too narrow an approach to take when studying e-learning environments. Evaluation research across the learning technology lifecycle We have argued above that investigations of learning technology investigations can involve a variable mixture of evaluation and research. We have also argued that the desired outcomes of e-learning evaluation research vary according to the stage of the e-learning lifecycle being studied, as do the research activities undertaken, and we illustrate this in Table 3, which expands on Table 2. We suggest that evaluation research of e-learning is an ongoing cyclical process which is closely related to the cycle of development of an e-learning environment (Bannan-Ritland, 2003; Nieveen, McKenney & van den Akker, 2006; van den Akker, 1999; Wang & Hannafin, 2005). Moreover, the characteristics of e-learning evaluation research are different at differing stages of the e-learning lifecycle. This notwithstanding, evaluation research at each stage should be based on a theoretical view of learning and/or contribute to the development of a theoretical view of learning. The first two columns of Table 3 identify the cycle of the e-learning lifecycle. The third column repeats the development activity for each stage. In the remaining columns in Table 3, the evaluation and research activities appropriate to each stage are considered. We need to caution against interpreting this table too literally. E-learning is complex, and investigations of e-learning can’t be undertaken in a lock-step, formulaic way. Researchers may engage with the lifecycle at different stages, stages may bleed into one another, and approaches to evaluation research may be appropriate across stages. Table 3 explicitly places the learning environment, process and outcomes within the cyclical e-learning evaluation-research process. The role of the teacher is highlighted in early cycles in Table 3, where the focus is on the design of the e-learning environment itself. We interpret ‘teacher’ broadly, to include the range of people who may contribute to the design of an e-learning environment – other teachers, educational designers, content experts, etc. Learners are drawn in through initial usability testing in cycle 3. In subsequent cycles, as the e-learning environment is shown to function as designed, the focus shifts from the environment to how learners engage with the environment – their learning processes and outcomes. Consideration of these different characteristics enables us to derive several distinct evaluation-research forms, which are discussed in detail in Phillips, McNaught and Kennedy (2011). Baseline analysis A baseline analysis is analogous to the ‘analysis of the problem’ aspect in design-based research. It documents teaching and learning practice (Littlejohn & Pegler, 2007). This includes the characteristics of the educational context, the nature of the institution and policy context, the structure of the degree program and any faculty- or department- specific information. It also describes the nature of the teaching and learning problem to be addressed. This includes a literature review and theoretical positioning of the research problem. The pedagogical and technological assumptions of the designers should also be clarified in the baseline analysis. Design evaluation The purpose of the design evaluation is to make judgments about the documented design of the learning environment. The design can be broken down into three distinct 1112 Australasian Journal of Educational Technology, 2012, 28(7) components: curriculum design (what students should learn), learning design (how students should learn), and the design of the e-learning artefact. In practice, this documentation may overlap with documentation produced as part of the baseline analysis, although there is a chronological distinction between the two. Table 3: Mapping of evaluation-research activities to the e-learning lifecycle Cycle Lifecyclestage Development activity Evaluation Research Role of theory and design principles 0 Analysis of problem Document the problem Baseline analysis Define teaching and learning problem based on scholarship 1 Design e-learning artefact Design e-learning artefact Design evaluation Design based on principles of e-learning best practice 2 Prototype e-learning artefact Develop e-learning artefact Project- management evaluation Formative evaluation of the e-learning artefact Reflecting on the characteristics of the e-learning environment Refine principles of e-learning best practice 3 Design e-learning environment and conduct pilot study Design e-learning environment Formative evaluation of the e-learning environment Refine principles of e-learning best practice 4 Refine e-learning environment and conduct full trial Refine e-learning environment Formative evaluation of the e-learning environment and processes Effectiveness research into learning processes Initial learning design principles 5 Evaluation research on mature system Confirm effectiveness of e-learning environment Summative evaluation Effectiveness research into learning processes and outcomes Refined learning design principles 6 Evaluation research on mature system Holistic understanding of how learners engage with the e-learning environment Summative evaluation Effectiveness research into learning processes and outcomes Refined learning design principles Project-management evaluation Since most e-learning developments are projects with fixed budgets and timelines, it is appropriate to make judgments about, and suggest improvements to, the conduct of that project. Project-management evaluation is primarily interested in processes, rather than outcomes, and it is primarily concerned with formative evaluation, although there are summative elements. Phillips, Kennedy and McNaught 1113 Formative evaluation Formative evaluation is an evaluation-research form which is appropriate at various stages of the e-learning lifecycle. The focus is on making judgments about, and suggesting improvements to, either an e-learning artefact or an e-learning environment and the learning tasks which are embedded in that environment. The primary interest of formative evaluation is to verify that the e-learning artefact, or e- learning environment, works in the way it was designed, and to identify any areas of improvement. Effectiveness research Once an e-learning environment is functioning as it was designed, effectiveness research becomes appropriate. This has an element of summative evaluation, making judgments about whether the e-learning environment is actually effective. However, the focus can now turn to the research side of the evaluation–research continuum, seeking to understand the learning processes that learners experience and the learning outcomes they achieve. Cyclical research approaches In the previous discussion, we have presented learning technology evaluation research as fundamentally concerned with the design and use of artificial phenomena within a learning context, broadly captured by what we have termed the e-learning lifecycle. We have also argued that different evaluation-research approaches are appropriate at different stages of the lifecycle. Some of these may be more appropriate to the technology, while others will explicitly address learning processes and outcomes. Overall this approach can be seen to be conceptually aligned with systematic, design- based approaches to research, such as those used in engineering (Burkhardt, 2006; Cobb, Confrey, diSessa, Lehrer & Schauble, 2003; Ross & Morrison, 1989; Salomon, 1991) rather than more analytic, scientific approaches often applied in the natural sciences. We have proposed a cyclical approach that explicitly maps evaluation-research activities to the design-and-development cycle of an e-learning artefact. This resonates with Bannan-Ritland’s ‘integrative learning design framework’, derived from approaches that integrate “instructional design, product design, usage-centred design, diffusion of innovations and educational research” (2003, p. 21). Bannan-Ritland’s work was developed to support a research approach called design-based research, a cyclical approach that originated in engineering and other design fields. It has emerged in recent years as a suitable approach to educational research (van den Akker, Gravemeijer, McKenney & Nieveen, 2006), in particular e-learning research (Herrington, Reeves & Oliver, 2010; Reeves, 2006). We see design-based research as just one of several approaches that can be usefully applied in the investigation of learning technology, but it is consistent with the arguments we have presented earlier about studies of e-learning requiring a mixture of evaluation and research; and with Stokes’ views on quest for understanding and consideration of use. “More than most other research approaches, [design-based] research aims at making both practical and scientific contributions” (van den Akker, 1999, p. 8). In this way, design-based research attempts to develop theory from analysing practice, but then uses theory to improve practice. It resonates more with development then research than with the research then development approach discussed earlier. 1114 Australasian Journal of Educational Technology, 2012, 28(7) Using and building theory in learning technology evaluation research In previous sections, we have seen two roles for theory: • Building theory as in a design-based approach. • Drawing on theory to support evaluation-research goals and questions across the e-learning lifecycle. As noted earlier, a criticism of e-learning developments has been the lack of theoretical frameworks to inform design. However, we also pointed out the difficulties in developing meaningful theories of human learning. Design-based research has a goal of adding to human knowledge, but addresses this issue by starting ‘small’, by attempting to develop relatively “humble” theories (Cobb, Confrey, diSessa, Lehrer & Schauble, 2003), addressing domain-specific learning processes which are “accountable to the activity of design” (p. 10). Theories in design-based research “are not ‘grand’ theories of learning ... Instead, they tend to emphasise an intermediate theoretical scope” (Cobb, Confrey, diSessa, Lehrer & Schauble, 2003, p. 11). They are intimately entwined with the design of learning environments (The Design-Based Research Collective, 2003), with the goal of providing a clear rationale about the suggestions and implications provided to practitioners. We will call these modest theories ‘proto-theories’. Design-based researchers tend to call their proto-theories design principles, with an emphasis on producing “heuristic guidelines to help others select and apply the most appropriate knowledge for a specific design task in another setting” (Nieveen, McKenney & van den Akker, 2006, p. 153). With the passage of time, and increasing understanding, design principles can potentially be generalised into theories (Edelson, 2006). Because proto-theories in design-based research are context-specific, it is very important to clearly specify the context (i.e. a baseline analysis) so that the findings of the study can be used in other, similar contexts. Kelly (2006) and the Design Based Research Collective (2003) have provided examples of use of design-based research in a range of contexts. If we now return to Table 3, we see in the last column the role of theory and design principles at different stages. At early stages of the e-learning lifecycle (baseline analysis and design evaluation), theory and design principles should be drawn upon to ensure that the needs analysis and design draws on principles of e-learning best practice. During subsequent stages of the e-learning lifecycle, the focus turns to generating and refining design principles, from both an e-learning best practice and from a learning theory perspective. Stand-alone studies The preceding sections have made a strong case for iterative cycles of evaluation research that lead to robust findings to inform both practice and theory in e-learning. However, there are many cases where circumstance and/or the research goal dictate a stand-alone approach to evaluation research, which can start at any stage of the e- Phillips, Kennedy and McNaught 1115 learning lifecycle. For example, there are many relatively mature learning environments that have been designed in the absence of any research activity. Also, we know that many e-learning projects have relatively short time-frames – either because of grant requirements or the need to have a finished product ready for the next term’s teaching. If an opportunity for useful research presents itself, then one should begin the process in the available context. This means adopting a pragmatic and strategic approach to e- learning evaluation research. If the research is worth doing and opportunities for data collection are available, then stand-alone studies can be very worthwhile. However, it is advisable not to think of projects in one-off terms, but to plan for further cycles of evaluation research as part of a sustained agenda. E-learning evaluation research does not have to be cyclical, but it should be able to demonstrate other aspects of rigour described in this paper. Conclusion In this review, we have attempted to shine new light on the problem of researching e- learning, and the role of theory in it. We have argued that systematic and structured inquiry in the area of learning technology is complex, but is essential for designing and developing optimal learning environments and building on the promise of learning technology. We have established that e-learning is an artificial phenomenon and several types of research are appropriate at different stages of the e-learning lifecycle, including consideration of whether a ‘manufactured’ artefact functions as designed. We argued that discovery research, such as that used when investigating natural phenomena, is too narrow an approach to take when studying learning technology. With natural phenomena researchers have to take them as they are; but research into designed phenomena is concerned with the design and functionality of the artefact which represents the phenomenon, as well as the behaviour of that phenomenon. These considerations led us to the idea that e-learning investigations involve a mixture of evaluation and research, and they need to focus on ‘use’ as well as understanding. Any one investigation can be placed anywhere along the evaluation–research continuum, depending on its desired outcomes. We mapped evaluation research against the e-learning lifecycle, and identified five evaluation-research forms that have different emphases when applied at different phases of the e-learning lifecycle. These are: baseline analysis, design evaluation, formative evaluation, effectiveness research, and project-management evaluation. We also argued that researchers should establish that the e-learning environment functions as designed before conducting summative, effectiveness research, and, while not the only approach, design-based research is a useful framework for guiding evaluation- research studies. While there is no general, unified educational theory, it is beneficial to be guided and build on previous research models, traditions and findings. However, since theory in learning technology is relatively weak, research efforts should go towards generating design principles and improved theoretical understanding in the field. Although it may not currently be possible, efforts should be made to generalise our theoretical understanding to a range of contexts. 1116 Australasian Journal of Educational Technology, 2012, 28(7) Acknowledgments We acknowledge the kind permission of Routledge for the extracts from Phillips, McNaught and Kennedy (2011) on which this paper has been based. References Bannan-Ritland, B. (2003). The role of design in research: The integrative learning design framework. Educational Researcher, 32(1), 21-24. http://dx.doi.org/10.3102/0013189X032001021 Becher, T. (1989). Academic tribes and territories: Intellectual enquiry and the cultures of disciplines. Buckingham: Society for Research into Higher Education. Biglan, A. (1973). The characteristics of subject matter in different subject areas. Journal of Applied Psychology, 57(3), 195-203. Boyer, E. L. (1990). Scholarship reconsidered - Priorities of the professoriate. New Jersey: Carnegie Foundation for the Advancement of Teaching. Burkhardt, H. (2006). From design research to large scale impact: Engineering research in education. In J. van den Akker, K. Gravemeijer, S. McKenney & N. Nieveen (Eds.), Educational design research (pp. 121-150). Abingdon, UK: Routledge. Cobb, P., Confrey, J., diSessa, A., Lehrer, R. & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9-13. http://dx.doi.org/10.3102/0013189X032001009 Cronbach, L. J. & Suppes, P. (Eds.) (1969). Research for tomorrow's schools: Disciplined inquiry for education. New York: Macmillan. Duncan, W. (Ed.) (1996). A guide to the project management body of knowledge. Upper Darby, PA: Project Management Institute. Edelson, D. C. (2002). Design research: What we learn when we engage in design. Journal of the Learning Sciences, 11(1), 105-121. http://www.jstor.org/stable/1466722 Edelson, D. C. (2006). Balancing innovation and risk: Assessing design research proposals. In J. van den Akker, K. Gravemeijer, S. McKenney & N. Nieveen (Eds.), Educational design research (pp. 100-106). Abingdon, UK: Routledge. England, E. & Finney, A. (1999). Managing multimedia (2nd ed.). Harlow, England: Addison Wesley Longman. Flyvbjerg, B. (2004). Five misunderstandings about case-study research. In C. Seale, G. Gobo, J. F. Gubrium & D. Silverman (Eds.), Qualitative research practice (pp. 420-434). London and Thousand Oaks, CA: Sage. Friesen, N. (2009). Re-thinking e-learning research: Foundations, methods and practices. New York: Peter Lang. Glass, G. V. & Worthen, B. R. (1971). Educational evaluation and research: Similarities and differences. Curriculum Theory Network, (8/9), 149-165. http://www.jstor.org/stable/1179200 Herrington, J., Reeves, T. C. & Oliver, R. (2010). A guide to authentic e-learning. New York and London: Routledge. Howell, G. T. (1992). Building hypermedia applications: A software development guide. McGraw Hill. Phillips, Kennedy and McNaught 1117 Hutchings, P. & Shulman, L. S. (1999). The scholarship of teaching: New elaborations, new developments. Change, 31(5), 10-15. http://www.carnegiefoundation.org/elibrary/ scholarship-teaching-new-elaborations-new-developments Jones, C., Zenios, M. & Griffiths, J. (2004). Academic use of digital resources: Disciplinary differences and the issue of progression. In Proceedings of the 4th Networked Learning Conference. Lancaster University, England, UK. http://www.networkedlearning conference.org.uk/past/nlc2004/proceedings/symposia/symposium9/jones_et_al.htm Kelly, A. E. (2006). Quality criteria for design research: Evidence and commitments. In J. van den Akker, K. Gravemeijer, S. McKenney & N. Nieveen (Eds.), Educational design research (pp. 107-118). Abingdon, U.K.: Routledge. Littlejohn, A. & Pegler, C. (2007). Preparing for blended e-learning. Abingdon, U.K.: Routledge. Nieveen, N., McKenney, S. & van den Akker, J. (2006). Educational design research: The value of variety. In J. van den Akker, K. Gravemeijer, S. McKenney & N. Nieveen (Eds.), Educational design research (pp. 151-158). Abingdon, U.K.: Routledge. Oliver, M., Harvey, J., Conole, G. & Jones, A. (2007). Evaluation. In G. Conole & M. Oliver (Eds.), Contemporary perspectives in e-learning research: Themes, methods and impact on practice (pp. 203- 216). Abingdon, Oxon: Routledge. Phillips, R. A., McNaught, C. & Kennedy, G. (2011). Evaluating e-learning: Guiding research and practice. New York and London: Routledge. Reeves, T. C. (1993). Pseudoscience in computer-based instruction: The case of learner control research. Journal of Computer-Based Instruction, 20(2), 39-46. Reeves, T. C. (2006). Design research from a technology perspective. In J. van den Akker, K. Gravemeijer, S. McKenney & N. Nieveen (Eds.), Educational design research (pp. 52-66). London: Routledge. Reeves, T. C. & Hedberg, J. G. (2003). Interactive learning systems evaluation. Englewood Cliffs, New Jersey: Educational Technology Publications. Richey, R. C., Klein, J. D. & Nelson, W. A. (2004). Developmental research: Studies of instructional design and development. In D. Jonassen (Ed.), Handbook of research for educational communications and technology (2nd ed., pp. 1099-1130). Mahwah, N.J.: Lawrence Erlbaum. Ross, S. M. & Morrison, G. R. (1989). In search of a happy medium in instructional technology research: Issues concerning external validity, media replication, and learner control. Educational Technology Research and Development, 37(1), 19-33. http://dx.doi.org/10.1007/BF02299043 Rowe, H. A. H. (1996). IT is failing to revolutionise the curriculum, because to date we have failed to evaluate its benefits in context. Paper presented at the Australian Computers in Education Conference, Canberra. Salomon, G. (1991). Transcending the qualitative-quantitative debate: The analytic and systemic approaches to educational research. Educational Researcher, 20(6), 10-18. http://dx.doi.org/10.3102/0013189X020006010 Shulman, L. S. (1988). Disciplines of inquiry in education: An overview. In R. M. Jaeger (Ed.), Complementary methods for research in education (pp. 3-17). Washington: AERA. Simon, H. A. (1969). The sciences of the artificial. Cambridge, MA: MIT Press. 1118 Australasian Journal of Educational Technology, 2012, 28(7) Stokes, D. E. (1997). Pasteur's quadrant: Basic science and technological innovation. Washington, DC: The Brookings Institution. The Design-Based Research Collective (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5-8. http://dx.doi.org/10.3102/0013189X032001005 Thornton, D. & Phillips, R. A. (1997). Evaluation. In The developer's handbook to interactive multimedia - A practical guide for educational applications (pp. 127-146). London: Kogan Page. Van den Akker, J. (1999). Principles and methods of development research. In J. van den Akker, R. M. Branch, K. Gustafson, N. Nieveen & T. Plomp (Eds.), Design approaches and tools in education and training (pp. 1-14). Dordrecht: Kluwer Academic. [verified 24 July 2012] http://projects.edte.utwente.nl/smarternet/version2/cabinet/ico_design_principles.pdf Van den Akker, J., Gravemeijer, K., McKenney, S. & Nieveen, N. (Eds.) (2006). Educational design research. Abingdon, U.K.: Routledge. Wang, F. & Hannafin, M. J. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research and Development, 53(4), 5-23. http://dx.doi.org/10.1007/BF02504682 Zaritsky, R., Kelly, A. E., Flowers, W., Rogers, E. & O’Neill, P. (2003). Clinical design sciences: A view from sister design efforts. Educational Researcher, 32(1), 32-34. http://dx.doi.org/10.3102/0013189X032001032 Authors: Associate Professor Rob Phillips, Educational Development Unit Murdoch University, South Street, Murdoch 6150, Western Australia Email: r.phillips@murdoch.edu.au Associate Professor Gregor Kennedy, Director of eLearning Centre for the Study of Higher Education, The University of Melbourne Email: gek@unimelb.edu.au Web: http://www.cshe.unimelb.edu.au/people/kennedy.html Professor Carmel McNaught, Centre for Learning Enhancement And Research The Chinese University of Hong Kong. Email: carmel.mcnaught@cuhk.edu.hk Web: http://www.cuhk.edu.hk/clear/people/Carmel.html Please cite as: Phillips, R., Kennedy, G. & McNaught, C. (2012). The role of theory in learning technology evaluation research. Australasian Journal of Educational Technology, 28(7), 1103-1118. http://www.ascilite.org.au/ajet/ajet28/phillips.html